Fathom v30 / styxx v7.26.0: Gold Anchors License Nothing -- label-free auditing of LLM judge panels, an instrument that refuses, and the measured failure of sanity checks

v30 (integrated): The anchor-threshold section now reports the shipped standard-test power (roughly 15 known-negatives for 90% power, via styxx.anchors.min_anchors_for_power) and adds a real-judge demonstration: on local Qwen panels grading TruthfulQA, a detectable all-judge shared blind spot arises only for a homogeneous weak panel (unanimous-wrong 0.64) and is caught by 3 known-label anchors at empirical power 0.959, matching the predicted 0.953; heterogeneous and strong panels show no blind spot. Certificate OATH-HELD, 96 verified / 0 contradicted. Gold Anchors License Nothing. LLM judge panels are the de facto measurement apparatus of modern AI evaluation, and the de facto validation of those judges is a set of blatant sanity checks (exact-duplicate pairs, direct negations, honeypots). We built a label-free auditor for judge panels with refusal semantics and measured operating characteristics, sealed its datasheet on simulation (nine of nine calibration gates), and ran the measurement the practice assumes away: across four task families on a real correlated panel, blatant gold checks license label-free coverage of 0/15 while the panel is flawless on the gold items, and the failure is silent when the violation is smooth. Ladder anchors drawn from the same generator repair it by pricing (13/13) or refusing (14/15 VOID). Two extensions: on a real heterogeneous panel (Gemini Flash-Lite + local Qwen) grading TruthfulQA, gold-anchor validity is prevalence-dependent (it misses the truth by up to 0.249 as the failure becomes common, while ladder anchors cover at every prevalence); and the impossibility is made quantitative (two unlabeled-identical panels are separated by a single known-negative at a 150.8x likelihood ratio, and caught with >90% power by ~30). The method can only VOID an eval, never bless one. Instrument: styxx.anchors (styxx 7.26.0, PyPI). Every number is receipt-bound and machine-verified (OATH 88/0).

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-07-24
DOI
https://doi.org/10.5281/zenodo.21522035
Primary Topic
Explainable Artificial Intelligence (XAI)
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
preprint

Fathom v30 / styxx v7.26.0: Gold Anchors License Nothing -- label-free auditing of LLM judge panels, an instrument that refuses, and the measured failure of sanity checks

Alexander Rodabaugh
Zenodo (CERN European Organization for Nuclear Research)
Explainable Artificial Intelligence (XAI)
preprint

Fathom v30 / styxx v7.26.0: Gold Anchors License Nothing -- label-free auditing of LLM judge panels, an instrument that refuses, and the measured failure of sanity checks

Alexander Rodabaugh
preprint en

Abstract

v30 (integrated): The anchor-threshold section now reports the shipped standard-test power (roughly 15 known-negatives for 90% power, via styxx.anchors.min_anchors_for_power) and adds a real-judge demonstration: on local Qwen panels grading TruthfulQA, a detectable all-judge shared blind spot arises only for a homogeneous weak panel (unanimous-wrong 0.64) and is caught by 3 known-label anchors at empirical power 0.959, matching the predicted 0.953; heterogeneous and strong panels show no blind spot. Certificate OATH-HELD, 96 verified / 0 contradicted. Gold Anchors License Nothing. LLM judge panels are the de facto measurement apparatus of modern AI evaluation, and the de facto validation of those judges is a set of blatant sanity checks (exact-duplicate pairs, direct negations, honeypots). We built a label-free auditor for judge panels with refusal semantics and measured operating characteristics, sealed its datasheet on simulation (nine of nine calibration gates), and ran the measurement the practice assumes away: across four task families on a real correlated panel, blatant gold checks license label-free coverage of 0/15 while the panel is flawless on the gold items, and the failure is silent when the violation is smooth. Ladder anchors drawn from the same generator repair it by pricing (13/13) or refusing (14/15 VOID). Two extensions: on a real heterogeneous panel (Gemini Flash-Lite + local Qwen) grading TruthfulQA, gold-anchor validity is prevalence-dependent (it misses the truth by up to 0.249 as the failure becomes common, while ladder anchors cover at every prevalence); and the impossibility is made quantitative (two unlabeled-identical panels are separated by a single known-negative at a 150.8x likelihood ratio, and caught with >90% power by ~30). The method can only VOID an eval, never bless one. Instrument: styxx.anchors (styxx 7.26.0, PyPI). Every number is receipt-bound and machine-verified (OATH 88/0).

Zenodo (CERN European Organization for Nuclear Research)
Peace, Justice and strong institutions
Explainable Artificial Intelligence (XAI)
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Fathom v30 / styxx v7.26.0: Gold Anchors License Nothing -- label-free auditing of LLM judge panels, an instrument that refuses, and the measured failure of sanity checks — Alexander Rodabaugh · Zenodo (CERN European Organization for Nuclear Research) (2026) | TGRS Research Map | TGRS