Large language model–based classification of adverse event severity: agreement with final human classification in a single-center study

Abstract Background Adverse event (AE) reporting systems are central to hospital safety monitoring, yet severity classification remains dependent on human review and may be subject to variability among reviewers. Large language models (LLMs) may support more consistent classification of safety reports. Objective To evaluate agreement between a large language model (LLM)-based classification system and adjudicated expert human classifications of adverse event (AE) severity, while characterizing variability in the human classification process. Methods We analyzed 286 adverse event reports from the institutional safety reporting system of a tertiary academic medical center. Two quality and safety experts independently classified each report according to an A–I severity scale aligned with the National Coordinating Council for Medication Error Reporting and Prevention (NCC MERP) framework; disagreements were adjudicated by a third senior reviewer to establish the final human classification. The same reports were processed by a prespecified LLM-based workflow whose configuration was fixed before evaluation, without training, fine-tuning, calibration, or prompt modification on the study dataset; when the workflow generated clarification questions, the recorded responses were used to produce a second classification. Exact agreement with the final human classification was treated as the primary descriptive agreement measure; one-category differences and larger discordances were considered separately. Results Initial exact agreement between the two human reviewers was 65.4% (187/286), with moderate inter-rater reliability (unweighted Cohen’s κ = 0.563; linear weighted κ = 0.740). The final LLM classification exactly matched the final human classification in 140 of 286 cases (49.0%); 125 (43.7%) differed by one category within the prespecified A–E near-match range, and 21 (7.3%) were larger discordances. Linear weighted Cohen’s κ was 0.728 (bootstrap 95% CI, 0.678–0.770). Among severe-harm cases (F–I, n = 48), 37 (77.1%) were exact matches and 11 (22.9%) were mismatches. Human reviewers spent approximately 61 cumulative hours on classification, whereas AI inference required 4 min and 47 s. Conclusions In this single-center study, an LLM-based system showed meaningful ordinal agreement with adjudicated expert severity classification, but clinically important discordances were concentrated toward under-classification. The findings support further evaluation of LLM-assisted severity classification as supervised decision support rather than autonomous adjudication. Prospective, multicenter, and independently validated studies are needed.