Every solubility model comes with an R2. Almost none of them tell you which split produced it, which quantity was predicted, or how many compounds were in the test set. Those three details decide whether the number means anything for your compounds.
This is how we evaluate CovaSyn's solubility prediction, what the numbers are, and where they stop being useful.
The one question that decides everything: which split?
A model's headline accuracy is mostly a statement about how the test set was built.
Random split.
Rows are shuffled and a fraction is held out. Analogues of the test compounds, meaning the same core with a different substituent and sometimes the same molecule at a different temperature, stay in training. The model interpolates between near-neighbours. Numbers look excellent. They tell you how the model behaves on chemistry it has already seen.
Scaffold split.
Compounds are grouped by their Bemis-Murcko core, and whole cores are held out. The test molecules have skeletons the model never saw. Harder, more honest, and closer to what you actually do: you are predicting a new series, not a new salt of an old one.
For our solubility prediction, the figure recorded at training time is R2 0.90 / RMSE 0.46 log. Re-evaluating that model's held-out scaffold split gives R2 approximately 0.92 with RMSE approximately 0.64 log across roughly 5,315 novel-scaffold rows. Different slices, not a contradiction, but the point stands: a number without its split is not interpretable.
We publish no random-split number for the deployed model. Evaluating the model on a random split would put its own training rows in the test set. That is memorisation, not accuracy.
The numbers
Held-out scaffold test, aqueous subset, n = 4,193. Units are log S in mol/L.
| Model | RMSE | MAE | R2 | Within 0.5 log |
|---|---|---|---|---|
| CovaSyn solubility prediction, held-out aqueous (n = 4,193) | 0.689 | 0.402 | 0.914 | 76.4% |
| ESOL (Delaney 2004), as published (n = 528) | 0.96 | 0.83 | 0.55 | not reported |
Two caveats belong right next to that table.
First, these are different test sets. The ESOL figures are the ones reported in the original paper (Delaney, *J. Chem. Inf. Model.* 2004, 44, 1000, 528-compound blind test). We did not recompute ESOL on our data. On a test set of larger, more modern molecules, the ESOL equation is far outside the chemistry it was fitted for, and re-running it there would produce an unfairly bad number. So this is literature context, not a same-test head-to-head. Treat it as "here is the classic reference point", not "we beat it by X".
Second, RMSE 0.689 log is the real error bar. MAE 0.402 is lower because the error distribution has a tail: most predictions are tight, a minority are badly wrong. MAE tells you the typical miss, RMSE tells you what the tail does to you. In practice, 76.4% of held-out predictions land within 0.5 log of the measured value: roughly a factor of three in concentration. That is a triage instrument, not an analytical result.
A worked example, and what the confidence interval is not
A prediction for aspirin in water at 298.15 K on 2026-07-25 returned log S = −1.536 (in mol/L), i.e. 5.24 mg/mL, with a 95% interval of [−1.539, −1.533] log and inside the applicability domain.
The measured value in our validation set is −1.72 log. So this prediction is off by roughly 0.18 log, which is comfortably inside the model's typical error.
Now look at that 95% interval: it is 0.005 log units wide. That is not a statement that the true value lies within 0.005 log of −1.536. It describes how stable the prediction is within the model, not how far it is from reality. Internal agreement is not accuracy. When you need a realistic error bar for a single prediction, use the held-out RMSE of 0.689 log, not the prediction's confidence interval.
Worth noting honestly: an earlier benchmark run records aspirin at −1.61 log, while the prediction on 2026-07-25 was −1.536 log. Predictions can shift between runs. If a number matters to a decision, record the prediction with structure, temperature and timestamp.
For context, here is the same fixed nine-compound validation panel (absolute error in log units):
- Aspirin 0.11, Paracetamol 0.11, Benzoic acid 0.23, Lidocaine 0.24
- Ibuprofen 0.32, Caffeine 0.38, Carbamazepine 0.49
- Salicylic acid 0.57, Naproxen 0.67
Nine compounds is far too small to be a benchmark. It is a sanity panel, and it is shown here as one: the defensible number is still the n = 4,193 scaffold holdout above.
Apparent solubility is not intrinsic solubility
This is the distinction that causes the most confusion when people compare logS models, and it is a definition problem, not a modelling problem.
Our prediction returns apparent solubility: the quantity present in most public solubility data, measured at whatever pH, buffer and salt form the source experiment used. It does not return intrinsic solubility S0, the solubility of the neutral species.
For ionizable compounds those two differ by 1 to 3 log units. Neutral molecules are fine: paracetamol predicted −1.14 against an intrinsic value of −1.03. Acids and bases are not.
We measured the consequence. On the public 2008 Solubility Challenge set, which reports intrinsic logS0, our model scored RMSE approximately 2.0 log even on compounds that were in its training data. That is what a units mismatch looks like. It is not evidence the model is broken; it is evidence you cannot compare an apparent-solubility model to an intrinsic-solubility benchmark and learn anything.
Two practical consequences:
1. If you need intrinsic S0 for a pre-formulation decision on an ionizable compound, do not read the apparent prediction as S0. Use a pH-dependent prediction or a pH-solubility profile with a measured or predicted pKa, and be explicit about which quantity you are quoting. 2. Be sceptical of any published solubility-challenge score that does not state whether the benchmark compounds were excluded from training. Roughly 82% of the 2008 Challenge compounds already appear in large public solubility datasets. A model that has seen 82% of the blind test is not being tested blind.
Applicability domain: the flag that decides whether to trust the number
Accuracy statistics describe the test set. They do not describe your molecule.
Every CovaSyn prediction carries a flag stating whether the compound lies inside the applicability domain. When it lies outside, the compound sits outside the feature space the model was fitted on, and the headline RMSE simply does not apply to it: the prediction is directional at best. In the solvent rankings we prepare for modern oncology compounds, out-of-domain solvents are flagged explicitly and are never used to carry a headline claim.
Prediction intervals widen for hard chemistry. On recent drug molecules the model reports 90% intervals of [−3.4, −3.0] for osimertinib and [−3.3, −3.2] for exatecan, but [−3.5, −1.4] for deruxtecan: a two-log-unit spread on a large, flexible ADC payload conjugate. That wide interval is the model doing its job. Read it, do not average it away.
What these numbers do not tell you
- Nothing about solid form. The underlying measurement data mixes polymorphs, hydrates and amorphous material. A polymorph change can move solubility by a log unit and the model will not see it coming.
- Nothing about kinetics. Equilibrium solubility is not dissolution rate. A compound can be adequately soluble and still fail a dissolution spec.
- Nothing about your formulation. Excipients, surfactants, cosolvent systems above modest fractions and complexation are outside scope.
- Nothing regulatory. These are decision-support and triage numbers for narrowing an experimental plan. They do not substitute for a validated solubility method and they do not go into a filing as generated.
- Nothing about a single compound's error. 76.4% within 0.5 log is a population statistic. It does not promise anything about the next molecule you type in.
Frequently asked questions
What is a good RMSE for a solubility prediction model?
For aqueous logS in mol/L, RMSE around 0.6 to 0.9 log on a scaffold-held-out test set is a realistic modern range. Our solubility prediction reaches RMSE 0.689, MAE 0.402 and R2 0.914 on 4,193 held-out aqueous rows with novel molecular cores. Numbers well below 0.5 log usually indicate a random split, where structural analogues of the test compounds remained in training.
Why is a scaffold split harder than a random split?
A random split leaves close analogues of every test compound in the training set, so the model interpolates between near-neighbours and scores high. A scaffold split holds out entire Bemis-Murcko cores, so the test molecules have skeletons the model never saw. Scaffold performance is the number that predicts how a model behaves on a genuinely new chemical series.
Does a narrow confidence interval mean the prediction is accurate?
No. In many models the reported interval only measures the internal spread, not how far the prediction is from the measured value. A CovaSyn aspirin prediction returned a 95% interval only 0.005 log wide while the actual error against the measured value was about 0.18 log. Use the held-out RMSE as your realistic error bar.
What is the difference between apparent and intrinsic solubility in these models?
Intrinsic solubility S0 is the solubility of the neutral species. Apparent solubility is what was measured at the experiment's pH, buffer and salt form, and it is what most public training data contains. For ionizable compounds the two differ by 1 to 3 log units. A model trained on apparent values cannot be scored fairly against an intrinsic-solubility benchmark such as the Solubility Challenge.
How should I compare a new model against ESOL?
Only on the same test set, or explicitly as published literature context. ESOL's original figures (Delaney 2004) are RMSE 0.96, MAE 0.83 and R2 0.55 on a 528-compound blind test. Re-running ESOL on a modern, larger-molecule dataset puts it far outside its fitted domain and understates it unfairly. State clearly which of the two you are doing.
What does the applicability domain flag actually change?
It tells you whether the reported accuracy statistics apply to your compound at all. If the flag is false, the molecule sits outside the model's fitted feature space and the headline RMSE is not transferable to it. Treat out-of-domain predictions as directional ranking only, check whether the prediction interval widened, and confirm anything decision-critical experimentally.
Related reading
Evaluation for this topic
We evaluate this for your lab.
- Solubility, pH, crystallization, antisolvent.