A machine-learning model for HFNC failure in older adults with COVID-19 showed inconclusive cross-site performance

To develop a machine-learning model for in-hospital high-flow nasal cannula (HFNC) failure in older adults with COVID-19 and to perform a preliminary cross-site check. We retrospectively analyzed 238 patients aged ≥ 65 years who received HFNC. Candidate algorithms were trained in 70% of the derivation cohort and selected using the remaining 30% hold-out set. The same hold-out set was used to report the selected model's AUC and is therefore termed the algorithm-selection set. The 13-predictor random-forest model (RF-13) was applied without modification to a 26-patient cross-site cohort. A five-predictor model (RF-5) was retained only as a post-hoc exploratory analysis. Baseline ROX-based indices and a post-hoc severity-excluded sensitivity analysis were examined. HFNC failure occurred in 161/238 patients (67.6%). RF-13 achieved an AUC of 0.812 (95% bootstrap CI, 0.700–0.908) in the algorithm-selection set and 0.527 (0.273–0.767) in the cross-site cohort. The wide cross-site interval included values consistent with both poor and potentially useful discrimination and did not permit a precise assessment of transportability. The post-hoc RF-5 yielded an AUC of 0.681 (0.542–0.810) in the algorithm-selection set and 0.788 (0.586–0.948) in the cross-site cohort; this imprecise reversal should not be interpreted as improved generalizability. APACHE II alone had a cross-site AUC of 0.700 (0.466–0.908). Decision-curve estimates for both RF-13 and RF-5 were driven by very few reclassified patients. Cross-site performance of the developed RF-13 model was too imprecise to establish or exclude transportability. The apparently higher RF-5 estimate was also imprecise and post hoc. These results do not support clinical use of either model without new, adequately powered multicenter validation.