Auditing the Auditor: Calibrated, Label-Free Measurement of Ungrounded Claims in LLM Incident Diagnosis

Research artefact for the paper "Auditing the Auditor: Calibrated, Label-Free Measurement of Ungrounded Claims in LLM Incident Diagnosis", 3rd Symposium on AI Trustworthiness and Risk Assessment (ATRACC), AAAI Fall Symposium Series 2026. Contents: the Telemetry Claim Adjudicator (TCA), a mechanical outcome-label-free instrument that adjudicates the telemetry claims an LLM agent cites when diagnosing a microservice incident; the diagnosis harnesses for six architectural arms over two benchmarks; the inference ledgers, keyed by a SHA-256 of each request, so every number in the paper traces back to a raw request and response pair; the four blind human audit rounds (352 claims) with the dossiers the rater saw, the withheld verdict keys, the per-claim verdicts and stated reasons, and the rubric fixed before the first claim was read; and the derived per-claim results and tables. Reproduction: ./reproduce.sh check runs in seconds with no data and no credentials. It re-derives every quantity the paper quotes from the shipped results through two independent paths, then checks the manuscript's formatting, its citations against Crossref and arXiv, and the README's own numbers. CLAIMS.md maps each claim in the paper to the command that produces it and states what cannot be reproduced. MANIFEST.sha256 fixes the contents of every file. Not included: the telemetry itself. Both datasets (RCA100 and AIOps2025, from Cai et al., arXiv:2606.29193) are licensed non-commercially and the RCA100 answer key carries an explicit no-redistribution notice, so src/fetch_aiops.py downloads them from the original source instead. No result depends on data held privately.