Science

CovaSyn is measured on a benchmark we did not design: MolecularIQ (Bartmann et al., ICLR 2026, Poster). Frontier models score 14 to 41 % on chemical structure analysis there; with CovaSyn MCP attached they reach 76 to 92 %. This page summarises methodology and results and provides the citation ready to copy.

Methodology

Independent, peer-reviewed benchmark

MolecularIQ was created by Bartmann et al. (Klambauer Lab, Institut fuer Machine Learning, JKU Linz) and accepted as a Poster at ICLR 2026. CovaSyn neither designed nor contributed to the benchmark; we measured our tools against it.

Symbolic verification

Every answer is verified symbolically against the stored solution: exact match or wrong. No language model acting as judge, no partial credit, no room for interpretation. That sentence matters more than any single number.

Scope

4 models, 3,540 verified tasks, 12,540 model responses. Data snapshot: 2026-05-17.

Reproducible

Dataset and evaluation code are public (links below). Every number shown here can be reproduced independently.

Results per model

Baseline means the same model without tools. The MCP column is the same model with the CovaSyn MCP server attached. Haiku 4.5 was evaluated on the full test split; the other three models on a proportionally stratified subset.

Model	Baseline	+ CovaSyn MCP	Δ	Lift
Claude Haiku 4.5	21.18 %	85.38 %	+64.20 pp	4.03×
Claude Opus 4.7	40.75 %	91.51 %	+50.76 pp	2.25×
OpenAI GPT-5.5	22.29 %	89.92 %	+67.63 pp	4.03×
Gemini 3.5 Flash	13.68 %	75.66 %	+61.98 pp	5.53×

Source: Bartmann C., Schimunek J., Ielanskyi M., Seidl P., Klambauer G., Luukkonen S. (2026). MolecularIQ: Characterizing Chemical Reasoning Capabilities Through Symbolic Verification on Molecular Graphs. ICLR 2026 (Poster). arXiv:2601.15279. Snapshot: 2026-05-17.

Benchmark summary: accuracy of all model configurations with and without CovaSyn MCP — Overall accuracy per model configuration. Image source: our own evaluation on the public MolecularIQ test split (Bartmann et al., ICLR 2026).

Cite this work

If you use the numbers on this page in a publication, a poster or an internal report, please cite the authors' MolecularIQ paper:

BibTeX

@inproceedings{bartmann2026moleculariq,
  title     = {MolecularIQ: Characterizing Chemical Reasoning Capabilities
               Through Symbolic Verification on Molecular Graphs},
  author    = {Bartmann, C. and Schimunek, J. and Ielanskyi, M. and
               Seidl, P. and Klambauer, G. and Luukkonen, S.},
  booktitle = {International Conference on Learning Representations (ICLR)},
  year      = {2026},
  note      = {Poster. arXiv:2601.15279},
  url       = {https://arxiv.org/abs/2601.15279}
}

Plain text

Bartmann C., Schimunek J., Ielanskyi M., Seidl P., Klambauer G., Luukkonen S. (2026). MolecularIQ: Characterizing Chemical Reasoning Capabilities Through Symbolic Verification on Molecular Graphs. ICLR 2026 (Poster). arXiv:2601.15279.

Paper, data, code

Paper on arXiv (abstract)https://arxiv.org/abs/2601.15279 Paper PDF on arXivhttps://arxiv.org/pdf/2601.15279 Dataset on Hugging Facehttps://huggingface.co/datasets/ml-jku/moleculariq-v0.0 Evaluation code on GitHubhttps://github.com/ml-jku/moleculariq

The PDF is hosted by arXiv, not on our servers. Dataset and code belong to the benchmark's authors.

The long version

Cost per question, category analysis, failure modes and the full set of charts live on the benchmark page.

See the full benchmark analysis →