Weighted calibrated metrics for perturbation-response prediction reward predicting the knockdown

Miller et al. reported that deep-learning models of genetic-perturbation responses can outperform uninformative baselines when scored with well-calibrated metrics, including a differential-expression-weighted mean squared error (WMSE) and weighted delta R². These metrics weight each gene by how strongly the perturbation changed it. In CRISPR interference (CRISPRi) screens the most strongly changed gene is often the targeted gene itself, and its knockdown is predictable from the perturbation label alone. We re-scored the authors' two unseen-gene CRISPRi benchmarks, Replogle K562 and Nadig HepG2, with their processed data and folds, reimplementing their scoring. Among test perturbations whose target was scored, the targeted gene received the largest weight in 53% and 36%, and it accounted for 28% and 19% of the dataset mean's WMSE. A baseline that copies the training mean and lowers only the targeted gene by the median training knockdown reached about 0.4 of the range between the authors' negative and positive controls, on both metrics and both datasets. This is about as far as scGPT, whose advantage over the dataset mean came entirely from the targeted gene. PRESAGE, scLambda and fMLP-GenePT, retrained with the authors' code, predicted almost none of the knockdown but outperformed the dataset mean on the other genes. On weighted R², PRESAGE outperformed the knockdown-aware baseline on both datasets, by 0.030–0.035 and 0.085–0.091. On WMSE, whose mean is dominated by knockdowns of a few highly expressed genes, no model outperformed it significantly. Given the baseline's value at the targeted gene, PRESAGE moved from 0.34 to 0.75 of that range on Replogle WMSE, and beyond the positive control on Nadig weighted R². Weighted metrics on CRISPRi data thus combine two separable achievements: predicting the knockdown, which needs no model of gene regulation, and predicting the downstream response. CRISPRi benchmarks should report target-excluded scores and on-target accuracy separately, and include a knockdown-aware baseline.