Fathom v17 / styxx v3.8.0: Multi-Signal Hallucination Detection at AUC 0.9012 on HaluEval-QA

Headline: styxx.guardrail.check() reaches AUC = 0.9012 on held-out HaluEval-QA (n=230), above every published baseline we compared against on the same task (SelfCheckGPT 0.71-0.79, KnowHalu 0.74, HaluCheck 0.82). System: Four-signal fusion via a logistic-regression meta-classifier fit on n=300 HaluEval-QA dev: knowledge_grounding (LR coef 8.21) - content-token coverage vs reference text_claim_risk (1.49) - length ratio, hedging, confidence markers entity_unverified_frac (1.43) - named entities not found in Wikipedia probe_confab (1.35) - residual-stream SAE confab probe (in-distribution HaluEval) intercept -3.46 Test AUC 0.9012 is stable across held-out folds; the earlier n=66 reading of 0.934 was small-sample noise. Companion release: styxx v3.8.0 on PyPI (pypi.org/project/styxx/3.8.0, MIT). Ships styxx.guardrail.check() (post-hoc) and styxx.generate_safe() (real-time self-halting generation - scores residuals every N tokens and stops if the confab probe crosses threshold). Three companion papers attached to this deposit: Universal Cognitive Basis: cross-vendor agreement in residual-stream directions (5-vendor leaderboard, mean rho = +0.472 across probes) Gradient-free capability amplification via residual steering (+7 pp TruthfulQA on Llama-1B with random-direction control) Multi-signal hallucination detection at AUC 0.90 on HaluEval-QA (focus of this release) Limits (stated up front): held-out AUC measured on HaluEval-QA only; cross-dataset validation on TruthfulQA / FactCC is queued but not in this release. knowledge_grounding dominates the LR fit - the signal's weight reflects HaluEval-QA's structure (reference passage available). When no reference is given, check() falls back to heuristic piecewise-linear fusion. Reproduction: pip install styxx==3.8.0. To reproduce 0.9012 from scratch, see benchmarks/hallucination_test/guardrail_calibrate.py in the git repo. License: CC-BY-4.0 (this deposit, papers, data) / MIT (styxx reference implementation). Repository: github.com/fathom-lab/styxx (tagged release v3.8.0, commit 55c5823).

Authors

Publication Details

Journal
Open MIND
Published
2026-04-22
DOI
https://doi.org/10.5281/zenodo.19701750
Primary Topic
Adversarial Robustness in Machine Learning
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
article

Fathom v17 / styxx v3.8.0: Multi-Signal Hallucination Detection at AUC 0.9012 on HaluEval-QA

Alexander Rodabaugh
Open MIND
Adversarial Robustness in Machine Learning
article

Fathom v17 / styxx v3.8.0: Multi-Signal Hallucination Detection at AUC 0.9012 on HaluEval-QA

Alexander Rodabaugh
article en

Abstract

Headline: styxx.guardrail.check() reaches AUC = 0.9012 on held-out HaluEval-QA (n=230), above every published baseline we compared against on the same task (SelfCheckGPT 0.71-0.79, KnowHalu 0.74, HaluCheck 0.82). System: Four-signal fusion via a logistic-regression meta-classifier fit on n=300 HaluEval-QA dev: knowledge_grounding (LR coef 8.21) - content-token coverage vs reference text_claim_risk (1.49) - length ratio, hedging, confidence markers entity_unverified_frac (1.43) - named entities not found in Wikipedia probe_confab (1.35) - residual-stream SAE confab probe (in-distribution HaluEval) intercept -3.46 Test AUC 0.9012 is stable across held-out folds; the earlier n=66 reading of 0.934 was small-sample noise. Companion release: styxx v3.8.0 on PyPI (pypi.org/project/styxx/3.8.0, MIT). Ships styxx.guardrail.check() (post-hoc) and styxx.generate_safe() (real-time self-halting generation - scores residuals every N tokens and stops if the confab probe crosses threshold). Three companion papers attached to this deposit: Universal Cognitive Basis: cross-vendor agreement in residual-stream directions (5-vendor leaderboard, mean rho = +0.472 across probes) Gradient-free capability amplification via residual steering (+7 pp TruthfulQA on Llama-1B with random-direction control) Multi-signal hallucination detection at AUC 0.90 on HaluEval-QA (focus of this release) Limits (stated up front): held-out AUC measured on HaluEval-QA only; cross-dataset validation on TruthfulQA / FactCC is queued but not in this release. knowledge_grounding dominates the LR fit - the signal's weight reflects HaluEval-QA's structure (reference passage available). When no reference is given, check() falls back to heuristic piecewise-linear fusion. Reproduction: pip install styxx==3.8.0. To reproduce 0.9012 from scratch, see benchmarks/hallucination_test/guardrail_calibrate.py in the git repo. License: CC-BY-4.0 (this deposit, papers, data) / MIT (styxx reference implementation). Repository: github.com/fathom-lab/styxx (tagged release v3.8.0, commit 55c5823).

Open MIND
Openalex Percentile: Top 5%
Adversarial Robustness in Machine Learning
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Fathom v17 / styxx v3.8.0: Multi-Signal Hallucination Detection at AUC 0.9012 on HaluEval-QA — Alexander Rodabaugh · Open MIND (2026) | TGRS Research Map | TGRS