RCI — the Revision Contingency Index
RCI = RAIR + RSR − 1 turns the two standard rates of appropriate reliance on advice into a single number: 0 when revising a decision is unrelated to whether the advice was right, +1 when an agent revises exactly when it should, −1 for the reverse. Reliance is currently reported as a pair of rates (RAIR, RSR; Schemmer et al., 2023), which has no chance level and cannot rank agents. Their sum minus one is the statistic known as Youden’s J and ΔP; this note names it for reliance on advice, states when it describes an agent, and shows that it must be computed per phase when behaviour changes over a task. Because it needs only four counts and does not depend on how often the advice is right, RCI is comparable across people, language models and AI agents. Applied to published studies, it reveals groups that revise against the evidence; applied to nine language models, it shows a responsive phase followed by one in which 8 of 9 models score exactly 0. A reference implementation is included.
Authors
- Anna Kovalchuk (ORCID: https://orcid.org/0009-0005-5143-812X)
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-11-16
- DOI
- https://doi.org/10.5281/zenodo.24268021
- Primary Topic
- Human-Automation Interaction and Safety
- Type
- preprint