Time-split validation and cross-platform transferability of an explainable MALDI-TOF MS machine-learning model for carbapenem-resistant Klebsiella pneumoniae screening

MALDI-TOF MS-based machine learning may shorten the time to detection of carbapenem-resistant Klebsiella pneumoniae (CRKP), yet performance estimates often rest on random splits and spectrum-level evaluation, and transferability across instruments is rarely tested. In a laboratory-information-system-audited single-center cohort of 2,399 K. pneumoniae isolates (363 CRKP; 2,836 VITEK MS spectra; 2024–2026), we trained a LightGBM classifier on 1,766 isolates from 2024 to 2025 and evaluated it on 633 isolates from 2026. The out-of-fold area under the ROC curve (AUC) was 0.909 (95% CI 0.885–0.932); on the 2026 cohort the calibrated AUC was 0.833 (0.776–0.885), with an expected calibration error of 0.024 and a Brier score of 0.068. Thresholds derived independently from the training cohort gave a sensitivity of 0.493 on the 2026 cohort at both the Youden and the NPV-oriented threshold, with an observed NPV of 0.935; the higher sensitivity obtained with thresholds chosen on the 2026 cohort itself was not reproducible. Applying the model unchanged to 1,190 isolates from three other platforms (Autof, Bruker, Quantof) resulted in loss of essentially all discriminative ability (AUC 0.450–0.543); because those datasets also differ in population and label definition, the loss cannot be attributed to platform alone. SHAP analysis identified a dominant m/z 4517.5–4519.5 cluster. A deployable offline tool reproduced the audited performance (AUC 0.826, ECE 0.022), and a monitoring analysis on 564 spectra drawn from the same period was consistent with it. The model is not a stand-alone screening test: deployment is limited to the platform on which it was developed and validated, and requires ongoing monitoring. The labeled dataset is deposited in MassIVE (MSV000103080) for independent benchmarking.