Explicit Applicability Domain Calculations Can Help Determine When Uncertainty Estimates Are Less Reliable

Quantifying the uncertainty associated with a QSAR prediction is hugely valuable. Conformal regression and Venn-ABERS have emerged as state-of-the-art uncertainty estimation methods for regression and classification QSAR models, respectively. However, their performance is limited when they are applied to compounds sampled from a different distribution to the data used to train the model and/or calibrate their uncertainty estimates. Previous studies have evidenced this when applying these methods to nonrandom train/test splits, e.g., temporal validation, cluster or scaffold splits. Building on these previous studies, we demonstrate that explicit applicability domain calculations, using only structural similarity, can help determine when these uncertainty estimates are less reliable for molecules encountered after model building. By less reliable, we mean the uncertainty estimates for out-of-domain predictions are less likely to reflect the empirically observed model residuals (regression) or probability of observing the predicted class experimentally (classification). After briefly comparing different methods using exemplar data sets, we extensively investigated the implications of computed applicability domain status for uncertainty estimation reliability using a k-nearest neighbors applicability domain approach (nUNC), in combination with Cross-Venn-ABERS Predictors (classification) or Aggregated Conformal Prediction (regression) uncertainty estimation across a wide range of public data sets. Because these are more representative of real-world applications, we focus on the results obtained on nonrandom test sets: temporal and cluster splits defined in previous modeling studies. We also present results for multiple temporal splits (time-splits) of classification and regression industrial data sets. In most cases, we found that nUNC was capable of distinguishing between molecules where the uncertainty estimates were, on average, more (inside the domain) vs less (outside the domain) reliable.