Clinical Prediction Model Performance Metrics Cluster at Arbitrary Thresholds and are Incorrectly Reported in High-Ranking Journals: An Observational Meta-Research Study

Background Previous research has shown that clinical prediction models have likely been 'hacked' to meet arbitrary performance thresholds. However, this work only assessed abstracts, could not identify models with a perfect AUC (area under the receiver operating characteristic curve) of 1, and did not examine other performance metrics. Methods We collected data from full texts and validated a large language model to extract data from a random sample of Q1 journal articles for 2025. We examined if hacking occurred in the AUC value, sensitivity and specificity, and checked if articles correctly reported the AUC value from corresponding receiver operating characteristic (ROC) curves. Results We identified 2,001 prediction model articles that reported 15,656 AUC values. We found a striking excess of results near perfect prediction and at multiples of five and ten for the AUC, sensitivity and specificity, indicating likely manipulated data, methods or publication bias. Around 1 in 40 AUC values were reported outside of their confidence intervals, and around 1 in 8 ROC curves did not match their reported AUC value. Conclusion We have shown that not only the AUC value, but the sensitivity and specificity are being hacked, likely to help the authors publish their paper at the cost of evidence for practice. Excess models that are not representative of their true performance may misguide clinicians, distort the literature, and make it harder to find prediction models that are useful for practice.