Explainer8 min readJuly 15, 2026

Applicability Domain in QSAR: When a Model Says I Don't Know

Q: What happens when a prediction is out of domain?

In CovaTox, `in_domain: false` automatically downgrades the prediction to the experimental launch tier, applies a confidence modifier of 0.1, and sets the risk level to "uncertain". The probability is still returned. For the EGFR inhibitor gefitinib, the DILI prediction of 0.90 came back with confidence 0.08 and tier experimental, because nearest-neighbour Tanimoto to the DILI training set was 0.00.

What the applicability domain in QSAR is, how Tanimoto, feature range and leverage methods define it, and how CovaTox downgrades out-of-domain predictions.

Oliver Kraft

CovaSyn

Applicability Domain in QSAR: When a Model Says I Don't Know

Every QSAR model returns a number for every molecule you feed it. That is the problem. A model trained on 1,000 marketed drugs will still hand you a confident-looking probability for a peptide macrocycle, a boron-containing warhead or a per-fluorinated surfactant it has never seen anything like. The applicability domain is the machinery that tells you which of those numbers you are allowed to act on.

What the applicability domain actually is

The applicability domain (AD) is the region of chemical space where a model's error estimate is still supported by evidence. It is OECD validation principle 3 for QSAR models, alongside a defined endpoint, an unambiguous algorithm, goodness-of-fit measures and a mechanistic interpretation. Without an AD statement, a QSAR result is not a validated prediction. It is an extrapolation with a decimal point.

Practically, the AD answers one question: how much of the training set is near this molecule? If the answer is "nothing within reach", the model is interpolating between points that do not exist, and the held-out accuracy you read off a benchmark table does not transfer.

The three AD methods you will actually meet

Most production systems combine two or three of these. They fail in different ways, which is why combining them is worth the effort.

Method	What it measures	Fails when
Nearest-neighbour similarity (Tanimoto on fingerprints)	Structural distance to the closest training compound	Similar structure, different mechanism (activity cliffs)
Feature range / bounding box	Whether each descriptor falls inside the training min-max	Every descriptor is in range individually but the combination is novel
Leverage (Williams plot, hat matrix)	Distance from the descriptor-space centroid, relative to a h* cutoff	Assumes a linear model; misleading for tree ensembles

CovaTox uses the first two and reports them separately, so you can see which gate failed. covatox_check_domain returns nearest_similarity (Tanimoto against the endpoint's training set, threshold 0.3), feature_coverage (fraction of descriptors inside the training range), and the two booleans tanimoto_ok and coverage_ok. Both must pass for in_domain: true.

Worked example: the same molecule, three verdicts

Take a representative EGFR inhibitor from the marketed-drug class (gefitinib, SMILES COc1cc2ncnc(Nc3ccc(F)c(Cl)c3)c2cc1OCCCN1CCOCC1). This is not a novel compound, it is a well-characterised reference. Below are verbatim covatox_check_domain outputs from the live gateway, run 2026-07-25.

Endpoint	in_domain	nearest_similarity	feature_coverage	confidence_modifier
Tox21 NR-AR	true	1.00	0.9635	0.9816
hERG	true	0.75	0.9650	0.8508
DILI	false	0.00	0.9808	0.1000

Same molecule, three different answers. The Tox21 NR-AR result is 1.00 because gefitinib is literally in the Tox21 screening library, so the nearest neighbour is itself. hERG sits at 0.75, comfortably inside a large and chemically diverse training set. DILI returns 0.00 similarity, so the Tanimoto gate fails even though 98 percent of its descriptors are in range. That is the bounding-box failure mode in the table above, caught by the second method.

What the platform does with that verdict

CovaTox attaches a launch tier to every prediction:

production - the model meets its calibrated benchmark target, with conformal intervals and AD calibrated.
gated - usable, but with a documented restriction.
experimental - shown for reference, not to be treated as reliable.

The escalation rule is blunt on purpose: `in_domain: false` downgrades any call to experimental, regardless of the endpoint's headline accuracy. A model does not get to keep its production badge on a molecule it has never seen.

Here is what that looks like on the same compound, from covatox_predict_endpoint:

hERG (in domain):

``probability: 0.15, predicted_class: 0, decision_threshold: 0.20 confidence: 0.5956, risk_level: medium, launch_tier: production conformal: prediction_set [0], set_size 1, coverage_level 0.90``

DILI (out of domain):

``probability: 0.90, predicted_class: 1, decision_threshold: 0.50 confidence: 0.08, risk_level: uncertain, launch_tier: experimental conformal: prediction_set [1], set_size 1, coverage_level 0.90``

Note what did *not* happen. The DILI probability is still 0.90. We do not hide it or round it away. What changes is the confidence, which drops from 0.5956 to 0.08 via the confidence_modifier of 0.1, the risk level, which becomes "uncertain" rather than "high", and the tier. A 0.90 out-of-domain DILI flag is a reason to look at the compound with a toxicologist. It is not a reason to kill the series.

The probability is never suppressed. Out of domain, confidence collapses to 0.08 and the tier drops to experimental, so a 0.90 flag triggers toxicologist review rather than a decision. Source: Verbatim covatox_predict_endpoint output for gefitinib DILI on the live CovaSyn gateway, run 2026-07-25.

Conformal prediction: the other half of the answer

Similarity tells you whether the model has seen this neighbourhood. Conformal prediction tells you how wide the answer has to be to keep a coverage guarantee. CovaTox returns a prediction_set at a stated coverage_level (0.90 above). The useful signal is the set size:

set_size: 1 - one class survives at 90 percent coverage. The model is committing.
set_size: 2 - both classes survive. The model cannot separate them at that coverage level. This is the honest "I do not know", and it is a different failure from being out of domain.

A molecule can be in-domain with a two-class set (the model has seen this chemistry and it is genuinely ambiguous) or out-of-domain with a one-class set (a confident-looking answer with no evidential support). Read both fields. They are not redundant.

Why headline accuracy does not settle this

The CovaTox scaffold-split holdout gives hERG an AUC of 0.856 on n_test = 7,500 and DILI an AUC of 0.909 on n_test = 72. The DILI number is higher. It is also the far weaker claim: 72 test compounds, and a training set narrow enough that a marketed EGFR inhibitor falls outside it. Any AUC quoted without a test-set size and a split type is decoration.

DILI's 0.909 rests on 72 compounds and a training set narrow enough to exclude a marketed EGFR inhibitor. hERG's 0.856 on 7,500 is the stronger evidence. Source: CovaSyn held-out benchmark: CovaTox scaffold-split holdout, n_test = 72 (DILI), 7,500 (hERG) and 1,453 (Tox21 NR-AR).

We report scaffold-split numbers as the headline because random splits leak. The gap is visible in our own data: Tox21 NR-AR is 0.830 scaffold versus 0.934 random on n_test = 1,453. Same model, same endpoint. The 0.934 is what you get when analogues of your test compounds sit in the training set. The AD check is what tells you which of those two regimes your specific query is in.

Honest limits

What the applicability domain does not tell you:

It does not validate the prediction. In-domain means the model has support here. It does not mean the model is right. hERG at 0.856 scaffold AUC still misclassifies compounds inside its domain.
Similarity is not mechanism. Two compounds at Tanimoto 0.9 can sit on opposite sides of an activity cliff. Fingerprint similarity is a proxy for mechanistic relevance, not a substitute.
Thresholds are conventions. The 0.3 Tanimoto cutoff is a defensible default, not a law of nature. Tighten it for regulatory-facing work.
Coverage of the AD reference set matters. Some CovaTox endpoints ship a full similarity reference set, others do not, and where it is missing the gate fails closed to out-of-domain. That is deliberate. An absent reference set must never read as "in domain".
None of this is a regulatory filing. CovaTox is triage and decision support. An ICH M7 or ICH S-series submission needs validated assays and expert review.

Frequently asked questions

What is the applicability domain in QSAR?

The applicability domain is the region of chemical space where a QSAR model's predictions are supported by its training data. It is OECD validation principle 3 for QSAR models. Outside that region, the model extrapolates and its reported accuracy no longer applies. A prediction without an applicability domain statement is not a validated result, and most regulatory reviewers will treat it that way.

How is the applicability domain calculated?

Three methods dominate. Nearest-neighbour similarity measures Tanimoto distance from the query to the closest training compound. Feature range checks whether each descriptor falls inside the training minimum and maximum. Leverage measures distance from the descriptor-space centroid against an h* cutoff. CovaTox combines Tanimoto (threshold 0.3) with feature coverage, and requires both to pass before flagging a compound in-domain.

What happens when a prediction is out of domain?

In CovaTox, in_domain: false automatically downgrades the prediction to the experimental launch tier, applies a confidence modifier of 0.1, and sets the risk level to "uncertain". The probability is still returned. For the EGFR inhibitor gefitinib, the DILI prediction of 0.90 came back with confidence 0.08 and tier experimental, because nearest-neighbour Tanimoto to the DILI training set was 0.00.

How does conformal prediction differ from an applicability domain check?

They answer different questions. The applicability domain asks whether the model has seen this chemistry. Conformal prediction asks how wide the answer must be to hold a stated coverage guarantee, typically 90 percent. A prediction set containing both classes means the model cannot separate them, even in familiar chemistry. Read both fields: a compound can be in-domain and ambiguous, or out-of-domain and falsely decisive.

Can I use a high AUC as proof a model will work on my compounds?

No. AUC is measured on a specific test set with a specific split. CovaTox reports DILI at 0.909 AUC on only 72 test compounds, and hERG at 0.856 on 7,500. The larger, lower number is the stronger claim. Always ask for the test-set size and whether the split was scaffold-based or random. Random splits inflate scores by leaking analogues into training.

What is a reasonable Tanimoto threshold for the applicability domain?

0.3 on Morgan-type fingerprints is a common working default and is what CovaTox uses. It is a convention, not a physical constant. For regulatory-facing or safety-critical decisions, raise it and require a tighter feature-coverage bound as well. Report the threshold you used alongside every prediction, because the same molecule flips domain status when the cutoff moves.