From Benchmark to Registry: A README Shortcut in a Learned npm Malware Detector

Static detectors of malicious npm packages report F1 and PR-AUC above 0.9 on public benchmarks. I test how well such scores carry over to real use. I train a gradient-boosted classifier (LightGBM) on 56 static features of package code and metadata, using the expert-reviewed benchmark of Guo et al. (ASE 2026), and evaluate it on random, campaign-grouped and time-ordered splits of that benchmark, on 2,893 malicious packages published in 2025–26, and on two samples of 5,000 random fresh npm uploads. On the benchmark, the model reaches PR-AUC 0.999. On real uploads, the threshold tuned for a 1% false-alarm rate flags 16.6% of packages, about 4,900 false alarms a day at npm's publish rate. The main cause is a shortcut the model learned: 99.8% of the benchmark's benign packages ship a README file, against 21% of its malware and 85% of fresh uploads. The model flags 79% of real packages that lack one. Random forest and logistic regression models learn the same shortcut, and adding a README to malware lets more of it through. Adding about 4,000 random fresh uploads to the benign training data, similar to the Amalfi detector's daily retraining, removes most of the problem: on the second sample, the retrained model catches 81% of 2025–26 malware at 0.7% false alarms (five-run means), against 72% when only the threshold is raised. Removing metadata features instead gives 2.8% false alarms on that sample, but catches less (75%). Hijacked releases of real libraries remain hard for the LightGBM models. These results confirm, with measurements for one standard npm detector, warnings already in the literature about curated benign sets and lab-only evaluation.