Auditing frozen chemical language model embeddings for drug-delivery screening: fingerprint baselines, random-initialization controls, and scaffold-memorization control tasks

Frozen embeddings from pretrained chemical language models (cLMs) are widely used as features for ADMET and drug-delivery prediction, often with a linear probe and little validation of what the representation actually contributes. We audit two small public cLMs (ChemBERTa-77M-MLM and ChemBERTa-77M-MTR, 3.4M parameters each) on three delivery-relevant endpoints from openly licensed sources: blood–brain barrier penetration (B3DB, CC0), PAMPA permeability (NCATS via PubChem), and Caco-2 permeability (OpenADMET, CC-BY-4.0). For each endpoint we compare frozen mean-pooled embeddings with (i) Morgan-fingerprint linear probes with train-only tuned regularization, (ii) the same architecture with random weights (five initialization seeds), and (iii) scaffoldconstant control tasks in the style of Hewitt and Liang that quantify how much a linear probe can score by memorizing scaffold identity. On observed means, tuned fingerprint probes are slightly ahead of frozen embeddings on all three endpoints (paired per-split differences of 0.016–0.032 in AUROC or R2, smaller than split-to-split variation); trained embeddings exceed random-initialization controls by 0.04–0.10; endpoint probes are selective above scaffold-memorization controls, and the memorization ceiling rises with layer depth in the primary MLM model. Decodability of bulk physicochemical descriptors (molecular weight, TPSA, fraction sp3, aromatic rings) is largely preserved under random initialization, indicating an architectural rather than learned origin for those signals; learned increments appear for logP, hydrogen-bond donors, and rotatable bonds. We quantify three practitioner pitfalls: pooling choice changes endpoint AUROC by up to 0.05; tuning the fin- gerprint baseline’s regularization changes it by up to 0.03 AUROC or 0.20 R2; and scaffoldsplit construction details change means by up to 0.04, exceeding seed standard deviations. The source datasets are public and cited; all curation scripts, pipeline code, and per-seed results are included in the accompanying package. The study is compute-bounded to CPU and small models, which limits the generality of the conclusions to that regime.