Applicability-Aware Reliability Scoring and Prediction of EMS Patient-Care Records for Trustworthy Smart Ambulance AI

This study presents the EMS Record Reliability Index (EMS-RRI), which measures how completely selected fields are documented in an emergency medical services encounter, when applicable. Using National Emergency Medical Services Information System (NEMSIS) data, we analyzed 200,000 records from 2024 and 500,000 from 2025. The basic score measured the percentage of applicable elements documented, while a domain-weighted version adjusted for differences in the number of elements across documentation domains. The two scores showed high agreement in both years. We labeled records at or below the basic EMS-RRI cutoff, defined by the 20th percentile of 2024 training scores, as low reliability. To predict this label, we developed and internally tested classifiers using 2024 records, then evaluated them on the full 2025 dataset without refitting. The final XGBoost model performed well, with ROC-AUC, PR-AUC, and F1 values of 0.9079, 0.8594, and 0.7637 on the 2024 internal test set and 0.8846, 0.8270, and 0.7319 on the 2025 test set. Most missed low-reliability records had scores near the cutoff, and performance varied across subgroups. In a separate retrospective analysis of high-acuity patient prediction, adding the EMS-RRI slightly improved the ROC-AUC and PR-AUC but showed no clear improvement in F1 score. Records flagged by the primary EMS-RRI cutoff did not have a higher downstream prediction error rate than unflagged ones. Overall, the EMS-RRI can identify incomplete documentation, but it does not verify clinical correctness. Independent and prospective validation is needed before operational use.