Self-Verification Lacks Self-Nature: Four Consecutive Measurements of an Author-Mutation Test Framework Failing Under Blind Fresh-Agent Audit
We built two small Python libraries (minpoint and ee_bot) whose stated purpose is to detect ungrounded groundings in a target computation, using mutation testing to verify that the detection mechanism has actual force. In four consecutive rounds — one round on minpoint and three rounds on ee_bot's successive versions v0.1, v0.2, v0.3 — the author's self-selected mutation test suite passed all tests (37, 35, 36, and 53 tests; 7, 8, and 14 mutations declared caught for the three ee_bot versions), then a blind fresh-agent adversarial audit found substantive attacks that survived the mutation suite in every case. In three of four rounds, the surviving attacks included a mechanical self-contradiction: the README made a general claim, and the code's behavior contradicted that claim at values not exercised by any test. We pre-registered a two-arm replication in a fourth round (patch vs. restructure); both arms produced one contradiction each, of the same shape but at different layers. We do not claim this pattern generalizes beyond the setting measured, and the outcome measure itself was selected post-hoc on the very invariance we report as a finding. The paper includes explicit scope-limitation (§5), pre-registration SHA-256 hashes for Round 4 committed before either audit ran, peer review outcomes (major-revisions then accept-with-minor-revisions), and the exact prompts used for both audits and peer reviews. This deposit contains: the paper; both Round 4 arm tarballs; the Round 4 pre-registration; reference implementations of v0.1, v0.2, v0.3; the Round 5 pre-registration (future work); and a README summarizing audit/peer-review results. All tarballs are self-contained (Python 3.11+, pytest, standard library only).
Authors
- Nobuki Fujimoto (ORCID: https://orcid.org/0009-0004-6019-9258)
- claude-opus-4-7) Claude (Anthropic
Institutions
- Institute for Anthropological Research (HR)
- Open Knowledge (United Kingdom) (GB)
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-09-13
- DOI
- https://doi.org/10.5281/zenodo.22731425
- Primary Topic
- Adversarial Robustness in Machine Learning
- Type
- article
- Field-Weighted Citation Impact
- 0.00