Discrimination transfers but calibration does not across fundus cameras in diabetic retinopathy screening

Abstract Deep-learning screening for referable diabetic retinopathy (DR) increasingly runs on cameras it was not developed on. We asked whether a referable-DR classifier trained on tabletop fundus photographs (BRSET) keeps its calibration — the agreement between predicted probability and observed frequency — as well as its discrimination when applied without adaptation to handheld images (mBRSET) and an independent external dataset (APTOS 2019). Across two networks and five training runs, zero-shot discrimination stayed within 0.03–0.04 AUROC of networks trained on each target (AUROC 0.89–0.96). Calibration did not: on handheld images, expected calibration error (ECE) was 0.06–0.07 without recalibration and 0.10–0.11 with the source-fitted calibrator (0.009 in-domain), and a threshold giving 90% sensitivity on the source detected only 42–47% of referable handheld images. On the handheld target, prevalence matching attributed about four-fifths ( $$79\pm 10\%$$ ) of the calibration loss to its higher disease prevalence; on the external target, prevalence and other dataset differences partly offset each other. Prevalence correction without labels did not restore calibration, whereas Platt or isotonic recalibration with 100–200 labelled target images did (ECE 0.03–0.04). Half-precision models preserved discrimination and calibration; 8-bit integer quantization reduced zero-shot AUROC by 0.07–0.14. New cameras therefore require recalibration on local labels.

Authors

Publication Details

Journal
Scientific Reports
Published
2026-11-14
DOI
https://doi.org/10.1038/s41598-026-78618-8
Primary Topic
Retinal Imaging and Analysis
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

Discrimination transfers but calibration does not across fundus cameras in diabetic retinopathy screening

Sairam Tabibu, Mugdha Abhyankar
3 links
Scientific Reports
Retinal Imaging and Analysis
article

Discrimination transfers but calibration does not across fundus cameras in diabetic retinopathy screening

Sairam Tabibu, Mugdha Abhyankar
article en

Abstract

Abstract Deep-learning screening for referable diabetic retinopathy (DR) increasingly runs on cameras it was not developed on. We asked whether a referable-DR classifier trained on tabletop fundus photographs (BRSET) keeps its calibration — the agreement between predicted probability and observed frequency — as well as its discrimination when applied without adaptation to handheld images (mBRSET) and an independent external dataset (APTOS 2019). Across two networks and five training runs, zero-shot discrimination stayed within 0.03–0.04 AUROC of networks trained on each target (AUROC 0.89–0.96). Calibration did not: on handheld images, expected calibration error (ECE) was 0.06–0.07 without recalibration and 0.10–0.11 with the source-fitted calibrator (0.009 in-domain), and a threshold giving 90% sensitivity on the source detected only 42–47% of referable handheld images. On the handheld target, prevalence matching attributed about four-fifths ( $$79\pm 10\%$$ ) of the calibration loss to its higher disease prevalence; on the external target, prevalence and other dataset differences partly offset each other. Prevalence correction without labels did not restore calibration, whereas Platt or isotonic recalibration with 100–200 labelled target images did (ECE 0.03–0.04). Half-precision models preserved discrimination and calibration; 8-bit integer quantization reduced zero-shot AUROC by 0.07–0.14. New cameras therefore require recalibration on local labels.

Scientific Reports
Good health and well-being
Retinal Imaging and Analysis
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon