Zero-shot mutation-effect predictors share a substitution-type bias: predicting per-mutation errors of ESM-2 for abstention

Zero-shot protein language models score every amino-acid substitution but do not say which predictions are likely to be wrong. I asked whether a model trained across assays, from information available at prediction time, can rank the within-assay errors of ESM-2 650M well enough for abstention. I trained a gradient-boosted model of each mutation’s absolute rank error on 162 ProteinGym development assays and evaluated it once, under an internally prespecified plan, on 49 test assays held out by 30% sequence-identity clustering. Its normalized area under the risk–coverage curve was 0.258, against 0.067 for the strongest baseline, the wild-type log-probability (mean paired difference 0.191 [0.143, 0.240]; better in 44 of 49 assays). Much of the gain corrects a bias by substitution type that ESM-2, ESM-1v and GEMME share: all three rank substitutions to cysteine, tryptophan, methionine and aromatic residues as more damaging than measured, and the ESM-2 pattern held in four of five assay selection types (post hoc). A cross-assay table of substitution-pair errors combined with the rank floor recovered 0.80 of the test-set gain over the baseline, and practitioners can use the deposited table directly. In a preregistered test on 13 proteins absent from training, the error model again beat the baseline (+0.095 [+0.010, +0.178]) and did so on a floor-robust metric computed within rank deciles (+0.057 [+0.005, +0.112]), but not on its whole-assay version (+0.002 [−0.094, +0.102]); its small increment over the pair table, real within ProteinGym, did not clearly transfer. A smaller preregistered test on new studies of the test proteins was inconclusive. The prespecified negative control failed and opening the test set was my decision; one study shared between training and test inflates the headline effect; the model did not improve top-50 selection; and it orders predictions rather than giving calibrated uncertainty. Manuscript and supplementary information are attached. Code, result files and the substitution-pair table are in the analysis repository.