Implications of Image-Level Duplication in Public Glaucoma Datasets

Importance: Medical artificial intelligence (AI) can be trusted only if validated in patients not used for development. In glaucoma AI, a few public datasets from 1 laboratory serve as those independent cohorts. Objective: To determine whether 3 public glaucoma datasets used for external validation and fairness evaluation are independent patient cohorts. Design, Setting, and Participants: This cross-sectional dataset audit and record-linkage analysis was conducted in July 2026 using the publicly distributed files of Harvard-GDP (1000 eyes), Harvard-GF (3300 eyes), and Harvard-FairVision (10 000 eyes), all from a single laboratory. Records were linked by exact visual field and demographic fingerprints, with chance modeled from the within-dataset collision rate and permutation. All 1302 candidate pairs underwent direct image comparison. Main Outcomes and Measures: Number of shared eyes and image-level identity of optical coherence tomography (OCT) scans distributed under different identifiers. Results: Harvard-GF and Harvard-FairVision matched on 1302 eyes against 1.9 expected by chance. Direct comparison confirmed 1031 (79.2%; 95% CI, 76.9%-81.4%) as the same 3-dimensional OCT volume (cosine similarity ≥0.9999; 88% of voxels exactly equal), at least 31% of the 3300 Harvard-GF eyes, a lower bound. Duplication was similar across the 3 Harvard-FairVision splits (78.5%-80.8%). Harvard-GDP and Harvard-GF shared 22 eyes matching exactly on all 52 total-deviation points and mean deviation, all in the Harvard-GF validation split; Harvard-GDP distributes no OCT volume, so its overlap rests on weaker evidence. No file carried a patient identifier, and no release disclosed the overlap; among the 64 of 105 citing studies whose full text we obtained, none reported it. Across 5 training runs, a classifier trained on 1 dataset and evaluated on the other gave an area under the curve of 0.90 (95% CI, 0.83-0.94) where 78.5% (95% CI, 74.0%-82.5%) of test eyes were training data, against 0.78 (95% CI, 0.72-0.83) out of sample (inflation, 0.12; 95% CI, 0.06-0.18). Inflation occurred in every racial subgroup and was larger where out-of-sample performance was lower (Spearman ρ = -0.57; permutation P = .02). Conclusions and Relevance: These findings suggest that the examined datasets were not independent cohorts. Treating them as independent benchmarks inflates reported performance and might mask fairness gaps through data leakage and nonindependent validation.

Authors

Institutions

Publication Details

Journal
JAMA Ophthalmology
Published
2026-11-12
DOI
https://doi.org/10.1001/jamaophthalmol.2026.5155
Primary Topic
Glaucoma and retinal disorders
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

Implications of Image-Level Duplication in Public Glaucoma Datasets

Yoshihisa Fukuhara, Souichi Oka, Ichigo Endo
JAMA Ophthalmology
Glaucoma and retinal disorders
article

Implications of Image-Level Duplication in Public Glaucoma Datasets

Yoshihisa Fukuhara, Souichi Oka, Ichigo Endo
article en

Abstract

Importance: Medical artificial intelligence (AI) can be trusted only if validated in patients not used for development. In glaucoma AI, a few public datasets from 1 laboratory serve as those independent cohorts. Objective: To determine whether 3 public glaucoma datasets used for external validation and fairness evaluation are independent patient cohorts. Design, Setting, and Participants: This cross-sectional dataset audit and record-linkage analysis was conducted in July 2026 using the publicly distributed files of Harvard-GDP (1000 eyes), Harvard-GF (3300 eyes), and Harvard-FairVision (10 000 eyes), all from a single laboratory. Records were linked by exact visual field and demographic fingerprints, with chance modeled from the within-dataset collision rate and permutation. All 1302 candidate pairs underwent direct image comparison. Main Outcomes and Measures: Number of shared eyes and image-level identity of optical coherence tomography (OCT) scans distributed under different identifiers. Results: Harvard-GF and Harvard-FairVision matched on 1302 eyes against 1.9 expected by chance. Direct comparison confirmed 1031 (79.2%; 95% CI, 76.9%-81.4%) as the same 3-dimensional OCT volume (cosine similarity ≥0.9999; 88% of voxels exactly equal), at least 31% of the 3300 Harvard-GF eyes, a lower bound. Duplication was similar across the 3 Harvard-FairVision splits (78.5%-80.8%). Harvard-GDP and Harvard-GF shared 22 eyes matching exactly on all 52 total-deviation points and mean deviation, all in the Harvard-GF validation split; Harvard-GDP distributes no OCT volume, so its overlap rests on weaker evidence. No file carried a patient identifier, and no release disclosed the overlap; among the 64 of 105 citing studies whose full text we obtained, none reported it. Across 5 training runs, a classifier trained on 1 dataset and evaluated on the other gave an area under the curve of 0.90 (95% CI, 0.83-0.94) where 78.5% (95% CI, 74.0%-82.5%) of test eyes were training data, against 0.78 (95% CI, 0.72-0.83) out of sample (inflation, 0.12; 95% CI, 0.06-0.18). Inflation occurred in every racial subgroup and was larger where out-of-sample performance was lower (Spearman ρ = -0.57; permutation P = .02). Conclusions and Relevance: These findings suggest that the examined datasets were not independent cohorts. Treating them as independent benchmarks inflates reported performance and might mask fairness gaps through data leakage and nonindependent validation.

JAMA Ophthalmology
Musashino University (JP)
Good health and well-being
Glaucoma and retinal disorders
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon