Retinal Nerve Fiber Layer OCT Quality Assessment: Comparing General-Purpose and Clinical AI Tools Using Global and Stepwise Prompting

As interest grows in multimodal large language models (LLMs) for ophthalmic imaging, their reliability must be established before clinical use. A key prerequisite is determining whether image quality permits interpretation. However, their ability to assess retinal nerve fiber layer (RNFL) optical coherence tomography (OCT) scan quality remains uncertain. We compared a general-purpose AI tool (ChatGPT-5.1) with a clinical AI tool (DoxGPT) for RNFL OCT report-quality assessment. Both evaluated 145 bilateral reports from 11 patients in an enriched quality-control sample using global and stepwise prompts. One ophthalmologist’s responses to the same predefined quality questions served as the clinical reference for calculating percent agreement, linearly weighted Cohen’s κ with patient-level cluster-bootstrap 95% confidence intervals (CIs), and prevalence-adjusted bias-adjusted κ (PABAK). Global prompting showed higher observed agreement than stepwise prompting: ChatGPT, 56% versus 47% (κ = 0.36 [95% CI, 0.23–0.49] versus 0.12 [0.08–0.18]); DoxGPT, 59% versus 48% (κ = 0.41 [0.28–0.54] versus 0.13 [0.03–0.23]). Component agreement was higher for signal strength (ChatGPT/DoxGPT: 99%/99%) and optic nerve head centering (87%/96%), and lower for dropout (41%/43%) and segmentation error (64%/65%). Both tools showed limited agreement with the ophthalmologist for overall RNFL report reliability. Their accuracy requires independent validation, supporting continued clinician review.