Same Error Rates, Opposite Winners: Reviewer summaries do not settle concept-intervention policy rankings: a pre-registered replication with real radiologists

Concept-intervention policies are usually compared under a simulated reviewer described by summary statistics, such as error rates per concept or overlap with model errors. We ask whether such summaries determine which of two policies is better. An earlier audit of one 57-patient ultrasound cohort found that opposite rankings of an adaptive policy (P3, entropy times recomputed diagnosis influence) and a fixed-entropy policy (P2) remain attainable after matching every policy's own consumed-answer totals. This paper reports a pre-registered second study. Its confirmatory part replaces synthetic answers with real radiologists' concept ratings from LIDC-IDRI: 622 nodule clusters in 423 patients, seven concepts, four reader fields and 25 cross-fitted concept-bottleneck models (pooled AUROC 0.787). The pre-specified primary result is insufficiency replicates: for every reader field, answer fields with identical consumption-matched summaries rank P3 and P2 either way (attainable ranges 44.4–48.4 pp wide), and no level of a nine-level summary ladder settles the ranking, including the strongest combined set (ranges still at least 20.2 pp wide). Under the real reader fields the two policies are nearly tied (−0.22 to +2.03 pp; every bootstrap interval includes zero). Uniform synthetic noise matched to each reader's per-concept error counts does not reproduce this; it makes P3 look 4.67–5.86 pp worse. Noise placed within model-error signature cells does reproduce it, and real reader errors are concentrated where the models err (pooled odds ratio 3.69). Exploratory extensions in the original cohort reproduce the earlier results exactly, show that finer summaries can settle the ranking there, show that the audit certifies in some settings but not where we predicted, and show that the ambiguity grows rather than shrinks with cohort size.

Authors

Institutions

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-19
DOI
https://doi.org/10.5281/zenodo.24374127
Primary Topic
Explainable Artificial Intelligence (XAI)
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Same Error Rates, Opposite Winners: Reviewer summaries do not settle concept-intervention policy rankings: a pre-registered replication with real radiologists

S. M. Abdulla Al Mamun
Zenodo (CERN European Organization for Nuclear Research)
Explainable Artificial Intelligence (XAI)
preprint

Same Error Rates, Opposite Winners: Reviewer summaries do not settle concept-intervention policy rankings: a pre-registered replication with real radiologists

S. M. Abdulla Al Mamun
preprint en

Abstract

Concept-intervention policies are usually compared under a simulated reviewer described by summary statistics, such as error rates per concept or overlap with model errors. We ask whether such summaries determine which of two policies is better. An earlier audit of one 57-patient ultrasound cohort found that opposite rankings of an adaptive policy (P3, entropy times recomputed diagnosis influence) and a fixed-entropy policy (P2) remain attainable after matching every policy's own consumed-answer totals. This paper reports a pre-registered second study. Its confirmatory part replaces synthetic answers with real radiologists' concept ratings from LIDC-IDRI: 622 nodule clusters in 423 patients, seven concepts, four reader fields and 25 cross-fitted concept-bottleneck models (pooled AUROC 0.787). The pre-specified primary result is insufficiency replicates: for every reader field, answer fields with identical consumption-matched summaries rank P3 and P2 either way (attainable ranges 44.4–48.4 pp wide), and no level of a nine-level summary ladder settles the ranking, including the strongest combined set (ranges still at least 20.2 pp wide). Under the real reader fields the two policies are nearly tied (−0.22 to +2.03 pp; every bootstrap interval includes zero). Uniform synthetic noise matched to each reader's per-concept error counts does not reproduce this; it makes P3 look 4.67–5.86 pp worse. Noise placed within model-error signature cells does reproduce it, and real reader errors are concentrated where the models err (pooled odds ratio 3.69). Exploratory extensions in the original cohort reproduce the earlier results exactly, show that finer summaries can settle the ranking there, show that the audit certifies in some settings but not where we predicted, and show that the ambiguity grows rather than shrinks with cohort size.

Zenodo (CERN European Organization for Nuclear Research)
University of Manchester (GB)
Explainable Artificial Intelligence (XAI)
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon