Interobserver variability in LOCS III cataract grading across experience levels: implications for AI training label quality

AIM: To evaluate the variability in cataract grading based on the Lens Opacities Classification System III (LOCS III) among ophthalmologists with different experience levels, and identify which type of annotated data may serve as high-quality labels for artificial intelligence (AI) model development. METHODS: Ophthalmologists were grouped by clinical experience (resident, senior, expert) and independently graded nuclear, cortical, and posterior subcapsular cataract (PSC) according to LOCS III. Consistency was analyzed at multiple levels: label consistency (presence/absence of opacity), grading consistency (strict and tolerant), and cross-subtype consistency using presence scores (agreement on cataract detection) and pairwise exact-match proportions (agreement on identical LOCS III grade). Intragroup reliability was assessed using intraclass correlation coefficients (ICCs), while intergroup and cross-subtype comparisons were conducted using Friedman and Kruskal-Wallis tests with Bonferroni-corrected post-hoc analyses. RESULTS: A total of 138 participants with a mean age of 66.64±10.09y (77 females) diagnosed with cataracts were included in the study, yielding 644 anterior segment slit-lamp photographs under mydriatic conditions. Each image was annotated independently by six ophthalmologists, including two residents, two attending-level seniors, and two experts. Experts achieved the highest intragroup agreement (ICC=0.780), while senior ophthalmologists and residents showed comparable moderate agreement. Intergroup comparison showed significantly higher label and grading consistency between senior-expert pairs than between resident-senior or resident-expert pairs, especially for cortical cataracts and PSC (P<0.05). Among subtypes, nuclear cataracts achieved the highest overall consistency, while cortical cataracts and PSC showed lower agreement, likely due to diffuse or obscured opacities and image quality variability (P<0.001). CONCLUSION: Consistency in LOCS III-based cataract grading varies with clinical experience and cataract subtype. Expert and senior ophthalmologists provide more stable annotations, and nuclear cataract images demonstrate superior interobserver agreement. These findings emphasize the value of expert consensus and subtype-specific data curation for improving AI-based cataract grading models.

Authors

Publication Details

Journal
International Journal of Ophthalmology
Published
2026-11-16
DOI
https://doi.org/10.18240/ijo.2026.12.01
Primary Topic
Intraocular Surgery and Lenses
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

Interobserver variability in LOCS III cataract grading across experience levels: implications for AI training label quality

Yu-Xuan Jia, Zhenyu Wang, Peilin Yue, Enjie Li et al.
3 links
International Journal of Ophthalmology
Intraocular Surgery and Lenses
article

Interobserver variability in LOCS III cataract grading across experience levels: implications for AI training label quality

Yu-Xuan Jia, Zhenyu Wang, Peilin Yue, Enjie Li, Xi-Jin Wu, Ran Zhang, Xue-Fei Ding, Xu-Dong Song, Jin-Da Wang, Jing Tao
article en

Abstract

AIM: To evaluate the variability in cataract grading based on the Lens Opacities Classification System III (LOCS III) among ophthalmologists with different experience levels, and identify which type of annotated data may serve as high-quality labels for artificial intelligence (AI) model development. METHODS: Ophthalmologists were grouped by clinical experience (resident, senior, expert) and independently graded nuclear, cortical, and posterior subcapsular cataract (PSC) according to LOCS III. Consistency was analyzed at multiple levels: label consistency (presence/absence of opacity), grading consistency (strict and tolerant), and cross-subtype consistency using presence scores (agreement on cataract detection) and pairwise exact-match proportions (agreement on identical LOCS III grade). Intragroup reliability was assessed using intraclass correlation coefficients (ICCs), while intergroup and cross-subtype comparisons were conducted using Friedman and Kruskal-Wallis tests with Bonferroni-corrected post-hoc analyses. RESULTS: A total of 138 participants with a mean age of 66.64±10.09y (77 females) diagnosed with cataracts were included in the study, yielding 644 anterior segment slit-lamp photographs under mydriatic conditions. Each image was annotated independently by six ophthalmologists, including two residents, two attending-level seniors, and two experts. Experts achieved the highest intragroup agreement (ICC=0.780), while senior ophthalmologists and residents showed comparable moderate agreement. Intergroup comparison showed significantly higher label and grading consistency between senior-expert pairs than between resident-senior or resident-expert pairs, especially for cortical cataracts and PSC (P<0.05). Among subtypes, nuclear cataracts achieved the highest overall consistency, while cortical cataracts and PSC showed lower agreement, likely due to diffuse or obscured opacities and image quality variability (P<0.001). CONCLUSION: Consistency in LOCS III-based cataract grading varies with clinical experience and cataract subtype. Expert and senior ophthalmologists provide more stable annotations, and nuclear cataract images demonstrate superior interobserver agreement. These findings emphasize the value of expert consensus and subtype-specific data curation for improving AI-based cataract grading models.

International Journal of OphthalmologyVol. 19(12)
Good health and well-being
Intraocular Surgery and Lenses
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon