A controlled framework for chemistry-shift evaluation in descriptor-based materials screening

Descriptor-based machine-learning models are widely used for property prediction and screening in cheminformatics and materials informatics, but random-split benchmarks do not by themselves establish reliability when target chemistry is absent from training. This issue is especially relevant in composition-only materials screening, where models use elemental identities and stoichiometric proportions without crystal-structure information. This work introduces a controlled framework for chemistry-shift evaluation that separates training-volume loss, target-regime displacement, and feature-space novelty. The framework combines family-held-out splits, size-matched random controls, target-distribution analysis, interval-calibration stress tests, detector complementarity, and selective abstention. Applied to XGBoost with Magpie descriptors on MatBench band gap ( N = 4604) and phonon peak frequency ( N = 1265), it shows that withholding a single element family increases MAE by up to +461% beyond matched controls. Conformal coverage falls from 89% to 41% for halogens at 90% nominal coverage, while uncertainty-guided abstention reduces retained MAE by 14–47%. Chemistry-shift evaluation therefore provides a reproducible protocol for determining when descriptor-based screening is transferable, unreliable, or suitable for uncertainty-guided triage.