Development of a machine learning ( ML ) model to predict 3‐month response to CGRP monoclonal antibodies in migraine: A pilot evaluation of physician‐ ML model interaction

BACKGROUND: Predicting individual treatment responses to anti-calcitonin gene-related peptide monoclonal antibodies (CGRP-mAbs) in migraine remains challenging. OBJECTIVE: To develop a machine learning (ML) model for predicting 3-month treatment response to CGRP-mAbs and to evaluate its practical clinical utility in improving physicians' predictive performance. METHODS: This retrospective prognostic model development was combined with a prospective, multi-reader, multi-case crossover reader study. We used retrospective data from 228 patients with migraine treated with CGRP-mAbs at the headache outpatient clinic in Dokkyo Medical University from April 2022 to March 2023. The data were divided into training (n = 182) and test (n = 46) datasets. Phase 1: An ML prediction model was developed using the training dataset with Random Forest and five-fold nested cross-validation. The model predicted achievement of a ≥50% reduction in monthly migraine days at 3 months, based on 27 baseline clinical variables. We tested the model's performance on the test dataset. Phase 2: Using the test dataset, five headache specialists and three general neurologists predicted treatment response without the model and then with the model's assistance. Prediction performance was compared between conditions. RESULTS: Phase 1: The model's performance on the test dataset was as follows: accuracy of 76.1% (95% confidence interval [CI], 62.1%-86.1%), sensitivity (recall) of 87.0% (67.9%-95.5%), specificity of 65.2% (44.9%-81.2%), precision of 71.4% (52.9%-84.7%), F1-score of 78.4% (64.0%-89.4%), and kappa of 0.52 (0.27-0.74) on the test dataset. Phase 2: The eight participating physicians predicted the treatment effectiveness. Without the model, their performance metrics were median accuracy of 65.2% [interquartile range 61.4%-70.1%], mean sensitivity (recall) of 68.5% ± standard deviation 11.6%, specificity of 62.5% ± 18.4%, precision of 66.3% ± 10.3%, F1-score of 66.7% [64.5%-71.9%], and kappa of 0.30 [0.23-0.40]. With the model, these metrics were accuracy of 73.9% [72.8%-76.1%], sensitivity (recall) of 79.9% ± 13.1%, specificity of 64.7% ± 13.6%, precision of 70.2% ± 8.1%, F1-score of 76.6% [74.4%-76.9%], and kappa of 0.48 [0.46-0.52], respectively. The accuracy, sensitivity, F1-score, and kappa were statistically improved. In the switching analysis, which assessed how physicians' predictions changed after receiving ML model assistance, incorrect-to-correct changes occurred in 12.8% (47/368, 8 physicians × 46 cases) of the pooled evaluations, whereas correct-to-incorrect changes occurred in 6.0% (22/368). The overall net switching benefit was +6.8%, and incorrect-to-correct changes occurred significantly more often than correct-to-incorrect changes (p = 0.003). CONCLUSIONS: Although external validation is necessary, our findings provide preliminary evidence that ML-assisted prediction may support the prediction of CGRP-mAb response.