Linkage Is Not Validity: Auditing the Actual Basis of a Belief

Linking a reason report to the right episode does not establish that it measures the reason actually used. Asking for a reason can prompt reconsideration, so a correctly linked report may describe a different inference. This paper develops a method for auditing a past basis. It models the probe separately from the original inference and shows when linked fallible indicators can be calibrated without a gold standard. Worked examples show how residual dependence and calibration drift undermine point identification and what can still be learned from bounded error or from the recorded overlap alone. A finite design criterion tells auditors which records must be available together for a given score; even all pairwise records can miss a three-way defect. An application to language-model faithfulness distinguishes an answer switch from actual reliance on a hint. The pump examples use synthetic data, and the simulator study is a proposed design rather than an empirical result.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-15
DOI
https://doi.org/10.5281/zenodo.24248948
Primary Topic
Explainable Artificial Intelligence (XAI)
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Linkage Is Not Validity: Auditing the Actual Basis of a Belief

Lorand Bruhacs
Zenodo (CERN European Organization for Nuclear Research)
Explainable Artificial Intelligence (XAI)
preprint

Linkage Is Not Validity: Auditing the Actual Basis of a Belief

Lorand Bruhacs
preprint en

Abstract

Linking a reason report to the right episode does not establish that it measures the reason actually used. Asking for a reason can prompt reconsideration, so a correctly linked report may describe a different inference. This paper develops a method for auditing a past basis. It models the probe separately from the original inference and shows when linked fallible indicators can be calibrated without a gold standard. Worked examples show how residual dependence and calibration drift undermine point identification and what can still be learned from bounded error or from the recorded overlap alone. A finite design criterion tells auditors which records must be available together for a given score; even all pairwise records can miss a three-way defect. An application to language-model faithfulness distinguishes an answer switch from actual reliance on a hint. The pump examples use synthetic data, and the simulator study is a proposed design rather than an empirical result.

Zenodo (CERN European Organization for Nuclear Research)
Explainable Artificial Intelligence (XAI)
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon