S1960 AI-Assisted Mayo Endoscopic Scoring in Ulcerative Colitis: A Comparative Analysis of GPT-4o and Claude 3.7 Sonnet

Introduction: The Mayo endoscopic subscore is a critical tool for assessing disease severity in Ulcerative Colitis (UC), guiding treatment and monitoring response. Traditional scoring relies on subjective endoscopist interpretation, leading to interobserver variability. This study evaluates GPT-4o and Claude 3.7 Sonnet, large language models (LLMs), in determining Mayo subscores from colonoscopy images, with an emphasis on potential clinical and low-resource utility. Methods: Colonoscopy images from 50 patients with confirmed UC were analyzed; all images were deidentified prior to analysis. Images from each colon segment (cecum, ascending, transverse, descending, sigmoid/rectum, and ileum) were extracted from reports and presented to GPT-4o and Claude 3.7 Sonnet in a chat interface, prompting the model to assign a Mayo subscore (0–3). Model-assigned scores were compared to those assigned by the performing endoscopist (gold standard). Accuracy was defined as the proportion of model-assigned scores matching the endoscopist’s. Segment-specific accuracy was calculated, and models were assessed on their ability to distinguish between low (Mayo 0–1) and high (Mayo 2–3) inflammation. Student’s t-tests were used to compare differences in accuracy between the models. Results: GPT-4o showed strong performance across all segments: descending colon (19/30), cecum (18/24), ascending (17/26), transverse (18/23), sigmoid/rectum (32/44), and ileum (19/22). Claude 3.7 Sonnet demonstrated lower accuracy: descending colon (9/30), cecum (6/24), ascending (13/26), transverse (12/23), sigmoid/rectum (24/44), and ileum (16/22). Overall, GPT-4o was significantly more accurate (123/169 [72.8%]) than Claude Sonnet 3.7 (80/169 [47.3%], P < 0.001), with significant differences in the descending colon (P = 0.009) and cecum (P = 0.000). Both models were similarly effective in differentiating low vs high inflammation (GPT-4o: 143/169 [84.6%], Claude: 130/169 [76.9%], P = 0.073). Conclusion: GPT-4o outperformed Claude Sonnet 3.7 in accurately assigning Mayo subscores, particularly in key segments. Both LLMs effectively distinguished low from high inflammation, suggesting potential to aid treatment stratification and decision-making—especially in settings lacking expert interpretation. Further development and validation are needed for clinical integration.