Predicting Critical Outcomes in Suspected Cardiopulmonary Emergencies Using Dispatch Narratives: Temporal Validation Study

Background: Early risk stratification in emergency medical services (EMS) is essential for patients presenting with acute cardiopulmonary symptoms, yet prehospital decision-making at the dispatch stage is often based on limited structured information. Free-text dispatch narratives may contain additional clinical signals, but their role in early risk assessment remains insufficiently characterized. Objective: This study aims to develop and temporally validate a natural language processing-assisted machine learning framework for early risk stratification using free-text EMS dispatch narratives and to evaluate its incremental value beyond conventional structured dispatch information. Methods: We conducted a population-based retrospective cohort study using EMS dispatch records from Nanning, China, between 2021 and 2025. Adult patients with suspected cardiopulmonary symptoms were identified based on predefined complaint keywords. After excluding nonmedical and incomplete records, 38,523 cases with available free-text narratives were included. To simulate real-world deployment, data from 2021 to 2024 (n=28,332) were used for model development, and 2025 data (n=10,191) served as an independent temporal test cohort. Dispatch narratives were processed using a natural language processing pipeline based on character-level n-grams and combined with structured variables (age, sex, call time) in a multimodal machine learning framework. The primary outcome was a composite prehospital critical outcome comprising death, clinical deterioration, or lack of response to initial treatment. Model performance was evaluated using area under the receiver operating characteristic curve (AUROC), area under the precision-recall curve, calibration, and decision curve analysis. Results: Among the 38,523 included patients, 12,476 (32.4%) experienced the primary composite outcome. The median age was 68.0 (IQR 54.0-79.0) years, and 23,019 (59.8%) patients were male. In the temporally independent 2025 cohort, an expanded structured baseline model achieved an AUROC of 0.681 (95% CI 0.669-0.693). Incorporation of narrative features improved performance (AUROC 0.803, 95% CI 0.792-0.814), with marginal additional gain from multimodal integration (AUROC 0.808, 95% CI 0.798-0.818). The multimodal model achieved an area under the precision-recall curve of 0.630 (95% CI 0.611-0.651), substantially exceeding the no-skill baseline defined by the outcome prevalence in the temporal test cohort (2464/10,191, 24.2%). Model performance remained consistent across age and sex subgroups, and calibration was acceptable (Brier score 0.1878). In a risk enrichment analysis, the top 10% (n=1019) of predicted high-risk cases accounted for 32.1% (n=791) of all critical outcomes, representing a 3.2-fold enrichment. Decision curve analysis indicated a higher net benefit compared with treat-all and treat-none strategies across a range of threshold probabilities. Conclusions: Free-text dispatch narratives contain clinically relevant information associated with early risk stratification in patients with suspected cardiopulmonary emergencies. Incorporating narrative-derived features into a structured modeling framework may complement existing EMS dispatch systems and support more informed decision-making prior to patient contact. Further external validation and prospective evaluation are warranted.