Inter‐observer variability is unlikely to explain regional differences in retinopathy of prematurity in Norway

PURPOSE: To test inter-observer variation of plus disease diagnoses among Norwegian ophthalmologists and treatment centres. METHODS: All ophthalmologists in Norway who make individual retinopathy of prematurity (ROP) treatment decisions were invited to grade retinal images using a web-based image grading platform. The images (n = 180) were a subset from the US Imaging and Informatics in Retinopathy of Prematurity (i-ROP) study and were either normal, with pre-plus disease, or plus disease. Agreement both between ophthalmologists and between the five different centres in Norway with a reference standard diagnosis (RSD) was measured through weighted kappa (k) and percent agreement (PA). Agreement with the US RSD was also measured. Each image was associated with an artificial intelligence-based Vascular Severity Score (VSS) ranging from 1 to 9, and the VSS agreement rate for each score was compared. RESULTS: All invited ophthalmologists (n = 15, of which six from the largest centre and two to three from each of the other four centres) agreed to participate. Agreement between ophthalmologists and the Norwegian RSD ranged from moderate (k = 0.458) to near perfect (k = 0.844), and from substantial (k = 0.679) to near perfect (k = 0.802) between consensus diagnosis at each centre and the Norwegian RSD. The agreement between centres and the US RSD ranged from moderate to substantial (κ = 0.561-0.729). CONCLUSION: Overall inter-observer variability in Norway was low. Such variability is therefore unlikely to explain regional differences in the frequency of treated ROP in Norway.