Development, Validation, and Silent Trial of a Predictive Model for Acute Care Use in Cancer Patients on Immunotherapy

Background: Immune checkpoint inhibitors (ICIs) are effective for treating melanoma, small-cell lung cancer, and other cancers. However, 54–76% of patients on ICIs experience immune-related adverse events (irAEs), which can lead to increased emergency department (ED) visits and hospital admissions. Proactively addressing these irAEs could reduce costly admissions and improve patient outcomes. The objective of this study was to develop, validate, and test via a silent trial a machine learning model for predicting ED and hospital admissions in patients receiving ICIs. Methods: The retrospective cohort study for model development and validation included 3962 adult patients from Duke University Health System who received ICIs between April 1, 2016, and July 1, 2021. The outcome of the model was ED or hospital admission within 3 weeks of ICI infusion. Performance measures included the area under the receiver operating curve (AUROC), area under the precision-recall curve (AUPRC), and sensitivity and positive predictive value (PPV) at various risk thresholds. A prospective silent trial of the model including 1,417 patients was conducted from February 16, 2023, to May 30, 2024. Performance measures included sensitivity, specificity, and PPV, as well as clinical reviewers’ agreement on risk categorization. Results: The model achieved an AUROC of 0.76 (95% CI: 0.73–0.79) and an AUPRC of 0.27 (95% CI: 0.20–0.34). At the high-risk threshold, sensitivity was 0.04 and PPV was 0.59; at the medium-risk threshold, sensitivity was 0.49, and PPV was 0.27. Key features included week of immunotherapy, encounter count, albumin, pulse, and thyroid stimulating hormone (TSH). In prospective validation, sensitivity was 24.6%, specificity was 94.3%, and PPV was 21.8% at the medium-risk threshold; at the high-risk threshold, sensitivity was 0.5% and PPV was 37.5%. Clinical reviewers agreed with 61.7% of risk categorizations. Conclusions; This study developed, validated, and tested a machine learning model to predict ED and hospital admissions following ICI infusion. This represents a promising strategy to proactively approach managing ICI therapy patients.