Benchmark8 min readJuly 15, 2026

Solubility Prediction Model Accuracy: How to Read It

Solubility prediction model accuracy explained: scaffold vs random splits, RMSE 0.689 on 4,193 held-out rows, and why apparent is not intrinsic solubility.

Oliver Kraft

CovaSyn

Solubility Prediction Model Accuracy: How to Read It

Every solubility model comes with an R2. Almost none of them tell you which split produced it, which quantity was predicted, or how many compounds were in the test set. Those three details decide whether the number means anything for your compounds.

This is how we evaluate CovaSolv, what the numbers are, and where they stop being useful.

The one question that decides everything: which split?

A model's headline accuracy is mostly a statement about how the test set was built.

Random split.

Rows are shuffled and a fraction is held out. Analogues of the test compounds - same core, different substituent, sometimes the same molecule at a different temperature - stay in training. The model interpolates between near-neighbours. Numbers look excellent. They tell you how the model behaves on chemistry it has already seen.

Scaffold split.

Compounds are grouped by their Bemis-Murcko core, and whole cores are held out. The test molecules have skeletons the model never saw. Harder, more honest, and closer to what you actually do: you are predicting a new series, not a new salt of an old one.

For CovaSolv, the training-time registry figure for the deployed ensemble (ensemble_v7) is R2 0.90 / RMSE 0.46 log. Reproducing that model's own held-out scaffold split deterministically and re-predicting it gives R2 approximately 0.92 with RMSE approximately 0.64 log across roughly 5,315 novel-scaffold rows. Different slices, not a contradiction - but the point stands: a number without its split is not interpretable.

We publish no random-split number for the deployed model. Evaluating ensemble_v7 on a random split would put its own training rows in the test set. That is memorisation, not accuracy.

The numbers

Deployed model ensemble_v7, held-out scaffold test, aqueous subset, n = 4,193. Units are log S in mol/L. Training data: BigSolDBv2.0 (CC-BY 4.0) and OChem.

Model	RMSE	MAE	R2	Within 0.5 log
CovaSolv, held-out aqueous (n = 4,193)	0.689	0.402	0.914	76.4%
ESOL (Delaney 2004), as published (n = 528)	0.96	0.83	0.55	not reported

Two caveats belong right next to that table.

First, these are different test sets. The ESOL figures are the ones reported in the original paper (Delaney, *J. Chem. Inf. Model.* 2004, 44, 1000, 528-compound blind test). We did not recompute ESOL on our data. On an OChem-heavy set of larger, more modern molecules, the ESOL equation is far outside the chemistry it was fitted for, and re-running it there would produce an unfairly bad number. So this is literature context, not a same-test head-to-head. Treat it as "here is the classic reference point", not "we beat it by X".

Second, RMSE 0.689 log is the real error bar. MAE 0.402 is lower because the error distribution has a tail: most predictions are tight, a minority are badly wrong. MAE tells you the typical miss, RMSE tells you what the tail does to you. In practice, 76.4% of held-out predictions land within 0.5 log of the measured value - roughly a factor of three in concentration. That is a triage instrument, not an analytical result.

A worked example, and what the confidence interval is not

A live call to covasolve_predict for aspirin in water at 298.15 K, run on 2026-07-25:

log S       = -1.536 mol/L
s_mg_ml     = 5.24
95% CI      = [-1.539, -1.533]
confidence  = 0.9998
applicability_domain = true

The measured value in our validation set is -1.72 log. So this prediction is off by roughly 0.18 log, which is comfortably inside the model's typical error.

Now look at that 95% interval: it is 0.005 log units wide. That is not a statement that the true value lies within 0.005 log of -1.536. It is the spread of the ensemble members - a measure of how much the constituent models agree with each other, not of how far they collectively are from reality. Ensemble agreement is not accuracy. When you need a realistic error bar for a single prediction, use the held-out RMSE of 0.689 log, not the ensemble CI.

The ensemble 95% interval was 0.005 log wide while the real error was 0.18 log. Ensemble agreement is not accuracy - quote the held-out RMSE of 0.689 log instead. Source: Live covasolve_predict call for aspirin in water at 298.15 K, run 2026-07-25, plus the CovaSolv ensemble_v7 held-out scaffold test (n = 4,193 aqueous rows).

Worth noting honestly: the archived benchmark asset for ensemble_v7 records aspirin at -1.61 log, while the live gateway returned -1.536 today. Serving paths and model versions drift. If a number matters to a decision, capture the run and its timestamp - covasolve_predict returns an audit block with canonical SMILES, temperature and timestamp for exactly this reason.

For context, here is the same fixed nine-compound validation panel (absolute error in log units):

Aspirin 0.11, Paracetamol 0.11, Benzoic acid 0.23, Lidocaine 0.24
Ibuprofen 0.32, Caffeine 0.38, Carbamazepine 0.49
Salicylic acid 0.57, Naproxen 0.67

Every compound in the fixed panel lands at or under the model's held-out RMSE of 0.689 log. Nine compounds is a spot check, not a benchmark. Source: Fixed nine-compound CovaSolv validation panel, absolute error in log units, as reported in this article.

Nine compounds is far too small to be a benchmark. It is a sanity panel, and it is shown here as one - the defensible number is still the n = 4,193 scaffold holdout above.

Apparent solubility is not intrinsic solubility

This is the distinction that causes the most confusion when people compare logS models, and it is a definition problem, not a modelling problem.

CovaSolv predicts apparent solubility: the quantity present in BigSolDB and OChem, measured at whatever pH, buffer and salt form the source experiment used. It does not predict intrinsic solubility S0, the solubility of the neutral species.

For ionizable compounds those two differ by 1 to 3 log units. Neutral molecules are fine: paracetamol predicted -1.14 against an intrinsic value of -1.03. Acids and bases are not.

We measured the consequence. On the public 2008 Solubility Challenge set - which reports intrinsic logS0 - CovaSolv scored RMSE approximately 2.0 log even on compounds that were in its training data. That is what a units mismatch looks like. It is not evidence the model is broken; it is evidence you cannot compare an apparent-solubility model to an intrinsic-solubility benchmark and learn anything.

Two practical consequences:

1. If you need intrinsic S0 for a pre-formulation decision on an ionizable compound, do not read the apparent prediction as S0. Use the pH-aware path (covasolve_ph_predict, covasolve_ph_curve) with a measured or predicted pKa, and be explicit about which quantity you are quoting. 2. Be sceptical of any published solubility-challenge score that does not state whether the benchmark compounds were excluded from training. In our own data, roughly 82% of the 2008 Challenge compounds already appear in BigSolDB or OChem. A model that has seen 82% of the blind test is not being tested blind.

Applicability domain: the flag that decides whether to trust the number

Accuracy statistics describe the test set. They do not describe your molecule.

Every covasolve_predict return carries an applicability_domain boolean. When it is false, the compound sits outside the feature space the model was fitted on, and the headline RMSE simply does not apply to it - the prediction is directional at best. In the solvent-ranking assets we generate for modern oncology compounds, out-of-domain solvents are flagged explicitly and are never used to carry a headline claim.

Prediction intervals widen for hard chemistry. On recent drug molecules the model reports 90% intervals of [-3.4, -3.0] for osimertinib and [-3.3, -3.2] for exatecan, but [-3.5, -1.4] for deruxtecan - a two-log-unit spread on a large, flexible ADC payload conjugate. That wide interval is the model doing its job. Read it, do not average it away.

What these numbers do not tell you

Nothing about solid form. The training data mixes polymorphs, hydrates and amorphous material. A polymorph change can move solubility by a log unit and the model will not see it coming.
Nothing about kinetics. Equilibrium solubility is not dissolution rate. A compound can be adequately soluble and still fail a dissolution spec.
Nothing about your formulation. Excipients, surfactants, cosolvent systems above modest fractions and complexation are outside scope.
Nothing regulatory. These are decision-support and triage numbers for narrowing an experimental plan. They do not substitute for a validated solubility method and they do not go into a filing as generated.
Nothing about a single compound's error. 76.4% within 0.5 log is a population statistic. It does not promise anything about the next molecule you type in.

Frequently asked questions

What is a good RMSE for a solubility prediction model?

For aqueous logS in mol/L, RMSE around 0.6 to 0.9 log on a scaffold-held-out test set is a realistic modern range. CovaSolv's deployed ensemble reports RMSE 0.689, MAE 0.402 and R2 0.914 on 4,193 held-out aqueous rows with novel molecular cores. Numbers well below 0.5 log usually indicate a random split, where structural analogues of the test compounds remained in training.

Why is a scaffold split harder than a random split?

A random split leaves close analogues of every test compound in the training set, so the model interpolates between near-neighbours and scores high. A scaffold split holds out entire Bemis-Murcko cores, so the test molecules have skeletons the model never saw. Scaffold performance is the number that predicts how a model behaves on a genuinely new chemical series.

Does a narrow confidence interval mean the prediction is accurate?

No. In ensemble models the reported interval usually measures how strongly the ensemble members agree with each other, not how far the ensemble is from the measured value. A CovaSolv aspirin prediction returned a 95% interval only 0.005 log wide while the actual error against the measured value was about 0.18 log. Use the held-out RMSE as your realistic error bar.

What is the difference between apparent and intrinsic solubility in these models?

Intrinsic solubility S0 is the solubility of the neutral species. Apparent solubility is what was measured at the experiment's pH, buffer and salt form, and it is what most public training data contains. For ionizable compounds the two differ by 1 to 3 log units. A model trained on apparent values cannot be scored fairly against an intrinsic-solubility benchmark such as the Solubility Challenge.

How should I compare a new model against ESOL?

Only on the same test set, or explicitly as published literature context. ESOL's original figures (Delaney 2004) are RMSE 0.96, MAE 0.83 and R2 0.55 on a 528-compound blind test. Re-running ESOL on a modern, larger-molecule dataset puts it far outside its fitted domain and understates it unfairly. State clearly which of the two you are doing.

What does the applicability domain flag actually change?

It tells you whether the reported accuracy statistics apply to your compound at all. If the flag is false, the molecule sits outside the model's fitted feature space and the headline RMSE is not transferable to it. Treat out-of-domain predictions as directional ranking only, check whether the prediction interval widened, and confirm anything decision-critical experimentally.