Beyond Acuity Accuracy: A Multidimensional Framework for Evaluating Patient-Facing Emergency Triage AI Tools
Patient-facing AI triage tools may support care-seeking and emergency department prioritization, but require evaluation beyond acuity accuracy alone. We developed SAFE-Triage, a framework integrating human review, automated metrics, LLM-as-judge evaluation, and simulation testing across clinical performance, patient interaction, and clinician-facing summary quality. We applied SAFE-Triage to 40 MIMIC-IV-derived emergency vignettes and a 10,030-case acuity analysis comparing prompted LLM triage (PLT) with rule-augmented prompted LLM triage (rPLT). PLT achieved higher emergency severity index (ESI) accuracy (57.5% vs 42.8%, $P < 0.001$), lower undertriage (13.1% vs 29.8%, $P < 0.001$), and higher clinician ratings across all domains. rPLT produced shorter and more stable outputs with higher automated readability scores. SAFE-Triage identified trade-offs between acuity accuracy, undertriage risk, communication quality, readability, and summary quality, supporting pre-deployment refinement and ongoing monitoring by showing where triage tools are efficient, need improvement, or need clinician oversight.
Authors
- Matthew A Bergens (ORCID: https://orcid.org/0000-0002-5395-2123)
- Nikita Paripati
- Henry P. Foote (ORCID: https://orcid.org/0009-0008-5973-2579)
- Chuan Hong (ORCID: https://orcid.org/0000-0001-7056-9559)
- An-Kwok Ian Wong (ORCID: https://orcid.org/0000-0001-5668-4251)
- Nicoleta Economou-Zavlanos (ORCID: https://orcid.org/0009-0000-4078-9809)
- Anand Chowdhury (ORCID: https://orcid.org/0000-0001-5703-113X)
- Mahmoud Alwakeel (ORCID: https://orcid.org/0000-0002-0565-4543)
- Kunyu Yu
- Jonathan Chong Kai Liew (ORCID: https://orcid.org/0009-0007-4094-2050)
- Ruio Heerschap
- Elisabeth Bond
- Monique Arnold
- Zhouhan Qian
- Rui Yang
- Yang Yang
- Kyle Naimoli
- João Ricardo Vissoci
- Nan Liu
- Diana Pimenta
- Jiaqi Wang
- Haoyuan Wang
Publication Details
- Journal
- arXiv (Cornell University)
- Published
- 2026-11-17
- DOI
- https://doi.org/10.48550/arxiv.2611.22045
- Primary Topic
- Artificial Intelligence in Healthcare and Education
- Type
- preprint