Fathom v17 / styxx v3.8.0: Multi-Signal Hallucination Detection at AUC 0.9012 on HaluEval-QA
Headline: styxx.guardrail.check() reaches AUC = 0.9012 on held-out HaluEval-QA (n=230), above every published baseline we compared against on the same task (SelfCheckGPT 0.71-0.79, KnowHalu 0.74, HaluCheck 0.82). System: Four-signal fusion via a logistic-regression meta-classifier fit on n=300 HaluEval-QA dev: knowledge_grounding (LR coef 8.21) - content-token coverage vs reference text_claim_risk (1.49) - length ratio, hedging, confidence markers entity_unverified_frac (1.43) - named entities not found in Wikipedia probe_confab (1.35) - residual-stream SAE confab probe (in-distribution HaluEval) intercept -3.46 Test AUC 0.9012 is stable across held-out folds; the earlier n=66 reading of 0.934 was small-sample noise. Companion release: styxx v3.8.0 on PyPI (pypi.org/project/styxx/3.8.0, MIT). Ships styxx.guardrail.check() (post-hoc) and styxx.generate_safe() (real-time self-halting generation - scores residuals every N tokens and stops if the confab probe crosses threshold). Three companion papers attached to this deposit: Universal Cognitive Basis: cross-vendor agreement in residual-stream directions (5-vendor leaderboard, mean rho = +0.472 across probes) Gradient-free capability amplification via residual steering (+7 pp TruthfulQA on Llama-1B with random-direction control) Multi-signal hallucination detection at AUC 0.90 on HaluEval-QA (focus of this release) Limits (stated up front): held-out AUC measured on HaluEval-QA only; cross-dataset validation on TruthfulQA / FactCC is queued but not in this release. knowledge_grounding dominates the LR fit - the signal's weight reflects HaluEval-QA's structure (reference passage available). When no reference is given, check() falls back to heuristic piecewise-linear fusion. Reproduction: pip install styxx==3.8.0. To reproduce 0.9012 from scratch, see benchmarks/hallucination_test/guardrail_calibrate.py in the git repo. License: CC-BY-4.0 (this deposit, papers, data) / MIT (styxx reference implementation). Repository: github.com/fathom-lab/styxx (tagged release v3.8.0, commit 55c5823).
Authors
- Alexander Rodabaugh
Publication Details
- Journal
- Open MIND
- Published
- 2026-04-22
- DOI
- https://doi.org/10.5281/zenodo.19701750
- Primary Topic
- Adversarial Robustness in Machine Learning
- Type
- article
- Field-Weighted Citation Impact
- 0.00