Benchmark and methodology

Independent benchmark

MolecularIQ was created by Bartmann et al. (Klambauer Lab, Institute for Machine Learning, JKU Linz) and published as a preprint on arXiv (arXiv:2601.15279). CovaSyn neither designed nor contributed to the benchmark; we measured our tools against it.

Symbolic verification

Every answer is verified symbolically against the stored solution: exact match or wrong. No language model acting as judge, no partial credit, no room for interpretation. That strictness makes the result robust against cherry-picking, and it matters more than any single number.

Tasks

We evaluated the test split of MolecularIQ with 3,540 tasks, restricted to question types that can be verified symbolically.

Models and scope

4 models: Claude Haiku 4.5, Claude Opus 4.7, OpenAI GPT-5.5 and Gemini 3.5 Flash, each without tools and with CovaSyn MCP. 12,540 model answers evaluated in total (one answer without and one with CovaSyn MCP per task and model); Haiku 4.5 on the full test split, the other three on a stratified sample of 910 questions per model to keep cost reasonable.

Tools

Chemistry primitives from the CovaBasicChem suite: deterministic cheminformatics that returns exact values, which the model then uses in its answer.

Snapshot and reproducibility

Data snapshot of 2026-05-17. Dataset and evaluation code are public and a CovaSyn account is free, so every number on this page can be recomputed.

@misc{bartmann2026moleculariq,
  title         = {MolecularIQ: Characterizing Chemical Reasoning Capabilities
                   Through Symbolic Verification on Molecular Graphs},
  author        = {Bartmann, C. and Schimunek, J. and Ielanskyi, M. and
                   Seidl, P. and Klambauer, G. and Luukkonen, S.},
  year          = {2026},
  eprint        = {2601.15279},
  archivePrefix = {arXiv},
  url           = {https://arxiv.org/abs/2601.15279}
}

Bartmann C., Schimunek J., Ielanskyi M., Seidl P., Klambauer G., Luukkonen S. (2026). MolecularIQ: Characterizing Chemical Reasoning Capabilities Through Symbolic Verification on Molecular Graphs. arXiv:2601.15279.

These are the building blocks of daily medicinal chemistry. A scaffold has to be identified exactly, a ring analysis must not drift into guessing, and a multi-constraint query such as “all branch points within two bonds of a halogen” has to resolve exactly.

The model is no longer automatically the bottleneck.

Teams paying about 0.03 US dollars per chemistry query today can move to about 0.008 US dollars at equal or better accuracy. That noticeably changes the cost of medicinal chemistry workflows.

The lift is model-independent.

Building an MCP layer does not lock you to Anthropic, OpenAI or Google. When the next frontier model arrives, the layer goes directly in front of it and the effect carries over.

The gap is disclosed.

For validation in regulated environments an open error distribution is worth more than a polished promise. Your QA team can derive risk assessment and acceptance criteria directly from the failure classes.