Structural error asymmetry and harm-weighted analysis of ChatGPT versus ICU Physicians in acid–base interpretation: a prospective observational study

Large language models (LLMs) have demonstrated potential in clinical reasoning tasks; however, their performance in real-world intensive care unit (ICU) acid–base interpretation remains insufficiently characterized, particularly in complex and mixed disorders. Most existing evaluations rely primarily on aggregate accuracy metrics without examining structural error patterns or the clinical severity of misclassification. In high-acuity ICU settings, under-recognition of physiological complexity may carry disproportionate safety implications. In this prospective observational study, arterial blood gas (ABG) data from 50 consecutive ICU patients were interpreted independently by ICU physicians and ChatGPT using a standardized prompt. Interpretations were harmonized into six predefined diagnostic categories and compared with a final reference diagnosis established by a blinded expert panel. Agreement was assessed using Cohen’s kappa and diagnostic accuracy metrics. Additional analyses included complexity-stratified evaluation, mixed-disorder sensitivity, multi-label component-level (metabolic and respiratory) detection, false reassurance risk assessment, harm-weighted misclassification modeling, bootstrap confidence intervals, and post-hoc power analysis. Overall categorical accuracy was 82% for ICU physicians and 72% for ChatGPT. Agreement was substantial in both groups (κ = 0.73 vs. 0.63). Paired comparison did not demonstrate a statistically significant difference in overall classification (p = 0.267), and post-hoc power analysis indicated limited ability to detect modest effect sizes (power = 0.22). However, stratified analyses revealed clinically meaningful structural differences. Sensitivity for mixed acid–base disorders was 0.96 for ICU physicians and 0.63 for ChatGPT. ChatGPT uniquely classified 16.7% of mixed cases as normal (false reassurance), whereas ICU physicians produced no false-normal classifications. Component-level analysis demonstrated lower respiratory component sensitivity for ChatGPT (0.88 vs. 1.00), contributing to under-recognition of physiological complexity. Harm-weighted misclassification modeling showed significantly higher clinical severity of errors for ChatGPT (mean difference 0.12; 95% CI 0.032–0.220; p = 0.026). While aggregate diagnostic agreement appeared broadly comparable, complexity-stratified and harm-weighted analyses demonstrated asymmetric error patterns with potential safety implications. These findings do not establish clinical equivalence and suggest that evaluation of AI diagnostic tools in critical care should extend beyond overall accuracy to incorporate safety-oriented and harm-weighted assessment frameworks.