Diagnostic Agreement Between a General-Purpose AI Model and Retinal Specialists in Color Fundus Photography—A Pilot Study

Background: Artificial intelligence (AI) has shown strong performance in disease-specific retinal screening tasks; however, its reliability in heterogeneous clinical diagnostic settings remains unclear. This study compared a general-purpose multimodal AI model with experienced retinal specialists in the interpretation of color fundus photographs (CFPs). Methods: In this pilot retrospective cross-sectional study, 66 CFPs were independently evaluated by a masked retinal specialist and an AI model (Google Gemini 2.5 Flash). Diagnoses were compared with those of the unblinded treating specialist. The comparison was inherently asymmetric, as the reference specialist had access to full clinical information, whereas the masked evaluators performed image-only assessment. Agreement was assessed using weighted percent agreement and Gwet’s AC2 with quadratic weights. Results: Substantial agreement was observed between the two human specialists (AC2 = 0.67). In contrast, agreement between the AI model and the reference specialist was low (AC2 = −0.58). Direct comparison between the masked specialist and the AI also showed limited reliability (AC2 = −0.38). Conclusions: In this pilot study, the evaluated AI model demonstrated limited agreement relative to a context-informed specialist reference. These findings support cautious interpretation of consumer-facing multimodal AI in open-ended retinal image assessment and warrant validation in larger, multicenter studies.