Self-Verification Lacks Self-Nature: Four Consecutive Measurements of an Author-Mutation Test Framework Failing Under Blind Fresh-Agent Audit

We built two small Python libraries (minpoint and ee_bot) whose stated purpose is to detect ungrounded groundings in a target computation, using mutation testing to verify that the detection mechanism has actual force. In four consecutive rounds — one round on minpoint and three rounds on ee_bot's successive versions v0.1, v0.2, v0.3 — the author's self-selected mutation test suite passed all tests (37, 35, 36, and 53 tests; 7, 8, and 14 mutations declared caught for the three ee_bot versions), then a blind fresh-agent adversarial audit found substantive attacks that survived the mutation suite in every case. In three of four rounds, the surviving attacks included a mechanical self-contradiction: the README made a general claim, and the code's behavior contradicted that claim at values not exercised by any test. We pre-registered a two-arm replication in a fourth round (patch vs. restructure); both arms produced one contradiction each, of the same shape but at different layers. We do not claim this pattern generalizes beyond the setting measured, and the outcome measure itself was selected post-hoc on the very invariance we report as a finding. The paper includes explicit scope-limitation (§5), pre-registration SHA-256 hashes for Round 4 committed before either audit ran, peer review outcomes (major-revisions then accept-with-minor-revisions), and the exact prompts used for both audits and peer reviews. This deposit contains: the paper; both Round 4 arm tarballs; the Round 4 pre-registration; reference implementations of v0.1, v0.2, v0.3; the Round 5 pre-registration (future work); and a README summarizing audit/peer-review results. All tarballs are self-contained (Python 3.11+, pytest, standard library only).

Authors

Institutions

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-09-13
DOI
https://doi.org/10.5281/zenodo.22731425
Primary Topic
Adversarial Robustness in Machine Learning
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
article

Self-Verification Lacks Self-Nature: Four Consecutive Measurements of an Author-Mutation Test Framework Failing Under Blind Fresh-Agent Audit

Nobuki Fujimoto, claude-opus-4-7) Claude (Anthropic
Zenodo (CERN European Organization for Nuclear Research)
Adversarial Robustness in Machine Learning
article

Self-Verification Lacks Self-Nature: Four Consecutive Measurements of an Author-Mutation Test Framework Failing Under Blind Fresh-Agent Audit

Nobuki Fujimoto, claude-opus-4-7) Claude (Anthropic
article en

Abstract

We built two small Python libraries (minpoint and ee_bot) whose stated purpose is to detect ungrounded groundings in a target computation, using mutation testing to verify that the detection mechanism has actual force. In four consecutive rounds — one round on minpoint and three rounds on ee_bot's successive versions v0.1, v0.2, v0.3 — the author's self-selected mutation test suite passed all tests (37, 35, 36, and 53 tests; 7, 8, and 14 mutations declared caught for the three ee_bot versions), then a blind fresh-agent adversarial audit found substantive attacks that survived the mutation suite in every case. In three of four rounds, the surviving attacks included a mechanical self-contradiction: the README made a general claim, and the code's behavior contradicted that claim at values not exercised by any test. We pre-registered a two-arm replication in a fourth round (patch vs. restructure); both arms produced one contradiction each, of the same shape but at different layers. We do not claim this pattern generalizes beyond the setting measured, and the outcome measure itself was selected post-hoc on the very invariance we report as a finding. The paper includes explicit scope-limitation (§5), pre-registration SHA-256 hashes for Round 4 committed before either audit ran, peer review outcomes (major-revisions then accept-with-minor-revisions), and the exact prompts used for both audits and peer reviews. This deposit contains: the paper; both Round 4 arm tarballs; the Round 4 pre-registration; reference implementations of v0.1, v0.2, v0.3; the Round 5 pre-registration (future work); and a README summarizing audit/peer-review results. All tarballs are self-contained (Python 3.11+, pytest, standard library only).

Zenodo (CERN European Organization for Nuclear Research)
Institute for Anthropological Research (HR), Open Knowledge (United Kingdom) (GB)
Openalex Percentile: Top 8%
Adversarial Robustness in Machine Learning
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.