Capturing reader-specific annotation biases in ordinal classification: a controlled simulation study

Objective. Inter-reader variability remains a major challenge in medical image classification, particularly for tasks involving ordinal labels, due to subjective interpretation and institutional biases. While deep learning is widely used to automate and standardize such classification tasks, including breast density assessment, handling reader-dependent annotation behavior remains an open problem. This study introduces a controlled setting to assess the extent to which such variability can be captured by a reader-adaptive model. Approach. This work builds on ReaderAdaptNet, a recently proposed reader-adaptive framework that models individual annotation styles through jointly learned reader-specific embeddings. Through a simulation study, it aims to deepen the understanding of this framework and to address the evaluation limitations imposed by small expert-annotated datasets, for which the underlying bias mechanisms are neither known nor controllable. Synthetic reader profiles are generated to reproduce realistic annotation behaviors, enabling a controlled analysis of the model's ability to adapt to diverse annotation biases. The effects of the embedding dimension and of the fusion strategy of image-reader features on classification accuracy are assessed. The structure of the embedding space is also analyzed. Main results. Results show that reader embeddings effectively capture identifiable, systematic biases, with moderate embedding dimensionality being sufficient. Systematic threshold shifts are recovered, whereas stochastic variability is not learnable. Conditioning image-reader fusion through feature-wise linear modulation or cross-attention mechanisms enables the correction of view-related discrepancies. The learned embedding space further reveals meaningful clustering of annotation profiles, reflecting similarity in reader behavior. Significance. Overall, the proposed synthetic framework identifies which aspects of inter-reader variability are learnable by ReaderAdaptNet and provides a controlled proxy for anticipating model behavior in clinical settings.

Authors

Institutions

Publication Details

Journal
Physics in Medicine and Biology
Published
2026-11-18
DOI
https://doi.org/10.1088/1361-6560/aec40c
Primary Topic
AI in cancer detection
Type
article
Field-Weighted Citation Impact
0.00

Funders

Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

Capturing reader-specific annotation biases in ordinal classification: a controlled simulation study

Clément Jailin, Laurence Vancamberg, Pablo Milioni de Carvalho, Isabelle Bloch A et al.
3 links
Physics in Medicine and Biology
AI in cancer detection
article

Capturing reader-specific annotation biases in ordinal classification: a controlled simulation study

Clément Jailin, Laurence Vancamberg, Pablo Milioni de Carvalho, Isabelle Bloch A, Elodie Ripaud
article en

Abstract

Objective. Inter-reader variability remains a major challenge in medical image classification, particularly for tasks involving ordinal labels, due to subjective interpretation and institutional biases. While deep learning is widely used to automate and standardize such classification tasks, including breast density assessment, handling reader-dependent annotation behavior remains an open problem. This study introduces a controlled setting to assess the extent to which such variability can be captured by a reader-adaptive model. Approach. This work builds on ReaderAdaptNet, a recently proposed reader-adaptive framework that models individual annotation styles through jointly learned reader-specific embeddings. Through a simulation study, it aims to deepen the understanding of this framework and to address the evaluation limitations imposed by small expert-annotated datasets, for which the underlying bias mechanisms are neither known nor controllable. Synthetic reader profiles are generated to reproduce realistic annotation behaviors, enabling a controlled analysis of the model's ability to adapt to diverse annotation biases. The effects of the embedding dimension and of the fusion strategy of image-reader features on classification accuracy are assessed. The structure of the embedding space is also analyzed. Main results. Results show that reader embeddings effectively capture identifiable, systematic biases, with moderate embedding dimensionality being sufficient. Systematic threshold shifts are recovered, whereas stochastic variability is not learnable. Conditioning image-reader fusion through feature-wise linear modulation or cross-attention mechanisms enables the correction of view-related discrepancies. The learned embedding space further reveals meaningful clustering of annotation profiles, reflecting similarity in reader behavior. Significance. Overall, the proposed synthetic framework identifies which aspects of inter-reader variability are learnable by ReaderAdaptNet and provides a controlled proxy for anticipating model behavior in clinical settings.

Physics in Medicine and Biology
Université Paris-Saclay (FR), Sorbonne Université (FR), General Electric (France) (FR)
Association Nationale de la Recherche et de la Technologie
AI in cancer detection
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon