Skip to content
CovaSyn
Intro call

Benchmark

Solubility prediction benchmarked: what the numbers say, and what they do not

On the public FastSolv benchmark our solubility model reaches an RMSE of 0.70 (SolProp) and 0.87 (Leeds) log units for compounds not in the training data. Every prediction carries a 90 % interval. What that means for solvent screening, and where a measurement still has to decide.

CovaSyn team7 min read

All Articles
Solubility prediction benchmarked: what the numbers say, and what they do not

Key takeaways

  • On the public FastSolv benchmark (Attia et al., Nat. Commun. 2025), our model, retrained only on the benchmark's own training data, reaches an RMSE of 0.70 log units on the SolProp test set and 0.87 on the Leeds test set. In both test sets, the test compounds do not occur in the training data.
  • The latest release is about 10 % more accurate for new compounds without measured reference data: in cross-validation the error fell from 0.89 to 0.80 log units.
  • Every prediction comes with a 90 % prediction interval and says when a compound lies outside the chemistry it knows well.
  • The interval is wide, about ±1.2 log units. That suits screening and solvent ranking. It does not replace a measurement.

The question behind the number

Early in development the typical case is a compound nobody has measured yet. Will it be a solubility problem in water and in process solvents, and which solvent should the first experiments use? A solubility prediction is useful there only if two things hold: it is accurate for compounds it has not seen, and it says how sure it is. This article reports both, with the qualifiers that belong to each number. All values are log10 of the solubility in mol/L; one log unit is a factor of 10 in concentration.

Public benchmark: FastSolv

FastSolv (Attia et al., Nat. Commun. 16, 7497, 2025) provides two test sets whose compounds do not occur in its training set. For this comparison our model was retrained from scratch on the benchmark's own training data only.

RMSE on the FastSolv SolProp and Leeds test sets with 95 % confidence intervals, next to the published FASTPROP and CHEMPROP values
Fig. 1. FastSolv test sets, RMSE in log units with 95 % CI. Published values taken from the paper; not a same-row comparison.
  • Leeds test set: RMSE 0.87 log units (95 % CI 0.83 to 0.92), compared with 0.95 published for FASTPROP and 0.99 for CHEMPROP.¹
  • SolProp test set: RMSE 0.70 log units (95 % CI 0.62 to 0.79); the published FASTPROP and CHEMPROP values are 0.83.¹

¹ Published values taken from the paper; not a same-row comparison.

New compounds without measured reference data

The second piece of evidence is internal cross-validation in which whole compounds are held out per fold, repeated on three independent fold assignments. It is not a public benchmark, and it says so in every caption.

RMSE for new compounds, previous and current release, for all solvents, organic solvents and water
Fig. 2. Cross-validation, new compounds without measured reference data. Previous vs. current release.
  • The latest release is about 10 % more accurate for new compounds without measured reference data: in cross-validation the error fell from 0.89 to 0.80 log units, consistently across three independent repeats.
  • In organic solvents the gain for such new compounds is about 13 % (0.86 to 0.74 log units in cross-validation).
  • In water the change is small (1.02 to 1.00).
  • For well-characterised compounds, accuracy is unchanged.

How sure is a single prediction?

Every prediction comes with a 90 % prediction interval; on held-out data it contained the true value 90.6 % of the time. For new compounds, the latest release's 90 % interval contains the true value 88 % of the time in cross-validation, up from 80 %, at the same width.

The interval is accurate on average but wide (about ±1.2 log units) and less reliable for very poorly soluble compounds. Below log S of about −4, roughly 0.1 mmol/L, it covers the true value only about 80 % of the time. For solvent mixtures the interval is marked as not calibrated, because coverage was measured for single solvents.

The tool also tells you when a compound lies outside the chemistry it knows well. Inside that domain, the error on held-out data was about 40 % lower (0.46 vs 0.76 log units). Outside means be more careful, not wrong: the flag is strict, and most held-out predictions fall outside it.

From a number to a process decision

A log S value is a means to an end. In process development the questions are which solvent, which charge and what yield. The same calculations therefore run further:

  • Solvent shortlists are filtered for liquid range, miscibility and ICH Q3C status before they are ranked. Solvents that Q3C does not list are marked as such.
  • Miscibility answers come from a curated table of 597 solvent pairs with a documented source for each verdict, including temperature-dependent pairs with a critical solution temperature, such as methanol and n-hexane.
  • Cooling crystallization: the tool proposes a charge and gives the expected yield with a 90 % interval, and it flags a charge that would leave most of the product in solution.
  • When a route cannot work (solvents immiscible, charge does not dissolve, cooling does not crystallize), the tool says so instead of returning a number.

The yields are equilibrium values. Nucleation, kinetics, polymorphs and particle size are outside what a calculation from structure can tell you.

What these numbers do not show

  • No independent validation. FastSolv is a public benchmark we scored ourselves; the published comparison values come from the paper, not from the same rows.
  • No guarantee for a single compound. An RMSE is a population statistic.
  • Not a replacement for a measurement. At about ±1.2 log units the interval is built for ranking candidates and solvents, not for a specification.
  • Known drugs are a sanity check, not a validation. Familiar compounds such as aspirin or paracetamol are in the training data, so a good match there says little about a new molecule.

Try it on your structure

Draw a molecule and get log S with its 90 % interval and domain check, solvent shortlists and crystallization yields. → CovaSketch

FAQ

How accurate is the solubility prediction for new compounds?

On the FastSolv benchmark the RMSE is 0.70 log units (SolProp) and 0.87 (Leeds) for compounds not in the training data. Published values for FASTPROP are 0.83 and 0.95; not a same-row comparison.

What does the 90 % interval mean?

On held-out data it contained the true value 90.6 % of the time. It is about ±1.2 log units wide and less reliable for very poorly soluble compounds.

Is the prediction good enough to replace a solubility measurement?

No. It is good enough to rank solvents and candidates and to avoid experiments on systems that cannot work.

Does it predict polymorphs or crystallization kinetics?

No. Crystallization yields are equilibrium values.

Methodology and data

FastSolv test sets SolProp (5,230 measurements, 101 compounds; the paper reports N = 5,700) and Leeds (1,294 measurements, 1,017 compounds), scored with the current release retrained on the FastSolv training set only; five training runs averaged; 95 % CI from 1,000 bootstrap resamples of the test compounds. The Leeds test set has no stated data licence and is used for scoring only. Cross-validation: whole compounds held out per fold, three independent repeats. Held-out interval and domain figures: 11,814 held-out measurements of compound families never used in training, previous model release. Training data include BigSolDB 2.0/2.2 (Krasnov et al., CC BY 4.0), AqSolDB (Sorkun et al., CC0), OPERA/PHYSPROP (US EPA, CC0), Biogen ADME (Fang et al.), ChEMBL (CC BY-SA 3.0) and the open data collections of Lang, Bradley & Acree (CC BY 4.0). Data snapshot: 2026-10-24.

Free PDF

Results you can rely on

Our beginner's guide for chemistry, pharma and biotech: how to tell an estimate from a calculation, seven questions to ask of every result and six questions to ask every provider. 8 pages.

Get the guide

Next step

Compute it instead of guessing.

30 minutes on Microsoft Teams with the CovaSyn team. If it is not a fit, we will say so.