Validation of ChatGPT-4o for detecting surgical site infections after colorectal surgery from electronic health records: A large-scale diagnostic accuracy study

BACKGROUND: Surgical site infection (SSI) surveillance is typically performed by infection control teams through manual review. This approach is resource-intensive and is prone to subjectivity. This study evaluates the performance of a large language model as a screening tool to reduce chart-review workload by flagging records likely to contain SSI evidence in clinical notes within the electronic health record (EHR) after colorectal surgery. METHODS: A retrospective diagnostic accuracy study was performed in elective colorectal surgery procedures (2010-2023). Narrative EHRs from the operative admission and early post-discharge period were automatically extracted, anonymized, and analyzed with a standardized prompt. ChatGPT-4o outputs (infected vs non-infected; SSI level) were compared with the results of manual surveillance based on full-chart review. Sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and accuracy were calculated from two-by-two contingency tables (95% CIs). RESULTS: A total of 1157 cases were analyzed. The reference standard identified an overall SSI prevalence of 11.32% (superficial incisional 3.28%, deep incisional 1.30%, organ/space 6.74%). For overall SSI detection, ChatGPT-4o-based classification achieved a sensitivity of 86.3% and a specificity of 85.7%, with PPV 43.5% and NPV 98%. Overall accuracy was 85.7%; (95% CI: 83.6-87.6), and agreement was substantial (PABAK 0.72). For superficial and deep incisional SSI, sensitivities were low (21.1% and 20%) despite high specificities (98.7% and 99.7%), whereas performance for organ/space SSI was similar to overall SSI (sensitivity 85.9%, specificity 84.6%). CONCLUSIONS: Under routine-prevalence conditions, ChatGPT-4o-based analysis of narrative EHR demonstrated good detection of overall SSI and organ/space-SSI, and a high ability to rule out SSI. These findings support its use as a screening layer within semi-automated surveillance workflows. Reliable detection of incisional SSI remains limited and may require improved documentation signals and/or integration of structured data sources. CLINICALTRIALS: gov: NCT06626399.