Evaluation of Differential Diagnosis Accuracy Using a Customized GPT-Based Artificial Intelligence Model in the Emergency Department (Preprint)

<sec> <title>BACKGROUND</title> Emergency departments (EDs) are high-pressure environments where rapid and accurate diagnosis is critical for patient outcomes. Artificial intelligence (AI)–based large language models (LLMs) are increasingly being explored as decision support tools to improve diagnostic accuracy, particularly when processing unstructured patient data. </sec> <sec> <title>OBJECTIVE</title> This study aimed to evaluate the diagnostic accuracy of differential diagnoses generated by a custom-trained with emergency medicine textbooks AI-based language model (Custom GPT) using unstructured symptom descriptions and physical examination findings of ED patients, and to assess associations with demographic and clinical variables. </sec> <sec> <title>METHODS</title> A retrospective, single-center study was conducted on 379 patients presenting to the ED. The model was trained using the Retrieval-Augmented Generation (RAG) method with a core emergency medicine textbook. For each case, the model generated one primary and four differential diagnoses. Primary diagnostic accuracy was compared with physician-confirmed diagnoses. Associations between accuracy and age, sex, triage category, organ system, and ED disposition were analyzed using statistical tests and correlation analysis. </sec> <sec> <title>RESULTS</title> The model achieved a primary diagnostic accuracy of 88.4%. Accuracy was highest in green triage patients (93.07%, p = 0.006) and lowest in red triage patients (75.31%, p &lt; 0.001). Accuracy was significantly lower in cardiovascular system cases (p = 0.002) and in hospitalized patients compared to those discharged (p = 0.002). No significant association with age or sex was observed. Correlation analysis revealed a weak but significant positive correlation for green triage (R = 0.141; p = 0.006) and weak but significant negative correlations for red triage (R = –0.213; p &lt; 0.001), hospitalization (R = –0.160; p = 0.002), and cardiovascular system cases (R = –0.162; p = 0.002). </sec> <sec> <title>CONCLUSIONS</title> A Custom GPT model demonstrated high diagnostic accuracy and potential as a clinical decision support tool in the ED. However, caution is warranted when applying such models to complex, high-acuity cases. </sec>