RCI — the Revision Contingency Index

RCI = RAIR + RSR − 1 turns the two standard rates of appropriate reliance on advice into a single number: 0 when revising a decision is unrelated to whether the advice was right, +1 when an agent revises exactly when it should, −1 for the reverse. Reliance is currently reported as a pair of rates (RAIR, RSR; Schemmer et al., 2023), which has no chance level and cannot rank agents. Their sum minus one is the statistic known as Youden’s J and ΔP; this note names it for reliance on advice, states when it describes an agent, and shows that it must be computed per phase when behaviour changes over a task. Because it needs only four counts and does not depend on how often the advice is right, RCI is comparable across people, language models and AI agents. Applied to published studies, it reveals groups that revise against the evidence; applied to nine language models, it shows a responsive phase followed by one in which 8 of 9 models score exactly 0. A reference implementation is included.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-16
DOI
https://doi.org/10.5281/zenodo.24268021
Primary Topic
Human-Automation Interaction and Safety
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

RCI — the Revision Contingency Index

Anna Kovalchuk
1 links
Zenodo (CERN European Organization for Nuclear Research)
Human-Automation Interaction and Safety
preprint

RCI — the Revision Contingency Index

Anna Kovalchuk
preprint en

Abstract

RCI = RAIR + RSR − 1 turns the two standard rates of appropriate reliance on advice into a single number: 0 when revising a decision is unrelated to whether the advice was right, +1 when an agent revises exactly when it should, −1 for the reverse. Reliance is currently reported as a pair of rates (RAIR, RSR; Schemmer et al., 2023), which has no chance level and cannot rank agents. Their sum minus one is the statistic known as Youden’s J and ΔP; this note names it for reliance on advice, states when it describes an agent, and shows that it must be computed per phase when behaviour changes over a task. Because it needs only four counts and does not depend on how often the advice is right, RCI is comparable across people, language models and AI agents. Applied to published studies, it reveals groups that revise against the evidence; applied to nine language models, it shows a responsive phase followed by one in which 8 of 9 models score exactly 0. A reference implementation is included.

Zenodo (CERN European Organization for Nuclear Research)
Human-Automation Interaction and Safety
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon