Comparison of anesthesiologist and Large Language Model recommendations for maintenance anesthetic dosing during upper extremity reconstruction without neuromuscular blockade

Maintaining an optimal depth of anesthesia is crucial for surgical success and patient safety. While large language models (LLMs) offer potential for clinical decision support, comparative studies evaluating their judgment against expert anesthesiologists remain scarce. This study aimed to compare the performance of three commercial LLMs (ChatGPT-4o, Copilot, and Gemini 1.5 Pro) against experienced anesthesiologists in determining maintenance doses of intravenous anesthetic agents during muscle relaxant-free upper extremity reconstructive surgery. This retrospective observational comparative study evaluated 198 patients undergoing upper extremity reconstructive surgery at Hazrat Fatemeh Hospital (March–April 2025). Demographic parameters, patient body weight, and clinical signs of light anesthesia were extracted retrospectively from perioperative medical records. Standardized prompts were evaluated across the three LLMs, and their suggested maintenance doses (defined as total cumulative bolus dose in mg administered during the entire maintenance phase) were compared against actual administered doses. Statistical analysis utilized paired t-tests, Intraclass Correlation Coefficients [ICC (2,1), Bland–Altman agreement analysis, Cohen’s kappa, and linear regression to assess agreement and diagnostic performance. Among 198 analyzed patients (89.4% male; mean age: 33.13 ± 11.53 years; mean weight: 81.19 ± 17.28 kg), intraoperative signs of light anesthesia occurred in 10.1% of cases ( n = 20). Anesthesiologists significantly escalated total maintenance doses in response to these clinical signs (23.55 mg vs. 15.58 mg; p < 0.001), whereas evaluated LLMs exhibited minimal dynamic adjustment (28.08 mg vs. 27.74 mg; p = 0.472). LLM-suggested doses correlated strongly with patient weight ( r = 0.881, p < 0.001; R-squared = 77.6%), compared to a moderate correlation in human decisions ( r = 0.363, p < 0.001; R-squared = 13.2%). Following patient parameter adjustments, Bland-Altman analysis demonstrated a substantial reduction in mean bias to -1.67 mg (SD = 5.87; 95% CI: -2.48 to -0.85 mg), with 95% limits of agreement ranging from − 13.16 to + 9.83 mg and an adjusted ICC (2,1) of 0.270. The raw ICC (2,1) for unadjusted LLM outputs was 0.041. The adjusted Bland–Altman analysis was performed using a post-hoc calibration factor derived from the mean human expert dose within the same study cohort; therefore, this adjustment does not reflect the intrinsic performance of the raw LLM outputs. Although LLMs demonstrated high specificity (100%) in identifying necessary dose increases, their sensitivity remained low at 36.0% (kappa = 0.496). Standalone commercial LLMs cannot replace clinical judgment in dynamic intraoperative environments. Their primary reliance on static parameters such as body weight and limited adaptation to real-time clinical signs can lead to sub-optimal dose titration. Future integration should focus on hybrid decision-support models, where medically fine-tuned AI acts as a real-time advisory tool under primary human supervision.