Reproducible Is Not Correct: Reflexive Gates and the Self-Descriptions We Ship to Language Models

Software increasingly ships descriptions of itself intended for a language model rather than a person: specifications, constrained-decoding grammars, JSON schemas, vendored context files, and the identity a server states to its host at handshake time. These artifacts are neither compiled nor executed against the thing they describe. The gate deployed over them, almost universally, is reflexive: regenerate the artifact and fail on a difference from the committed copy. We observe that such a gate establishes determinism and cannot, even in principle, establish truth — and that it is nonetheless sound wherever it is deployed, because the artifacts it guards are mechanically derived from their subject, and a derived artifact can be stale but never wrong. That precondition is never stated and never checked. When a generator retypes a fact instead of deriving it, the same gate is vacuous, and it looks identical in continuous integration. We measure what this costs. A census of all 24,131 servers listed on the Model Context Protocol registry, covering the 7,272 unique npm packages among them, finds that of the 5,052 packages whose self-reported identity can be resolved statically, 89.6% retype it rather than derive it, and 41.1% state a version that once tracked the package and has since been left behind — corroborated against each package's own release history, with independently versioned cases excluded. We add six instances across four independent third-party compilers, four of them verified by execution, and a depth study of one production language whose four self-descriptions were all wrong on first check. We claim none of the underlying concepts, all of which are established; we report the precondition, the artifact class, and the measurement. The subject system was substantially co-authored with language models, and we discuss what that implies for which guards matter.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-09-25
DOI
https://doi.org/10.5281/zenodo.22958002
Primary Topic
Software Engineering Research
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
preprint

Reproducible Is Not Correct: Reflexive Gates and the Self-Descriptions We Ship to Language Models

Chan Meng
Zenodo (CERN European Organization for Nuclear Research)
Software Engineering Research
preprint

Reproducible Is Not Correct: Reflexive Gates and the Self-Descriptions We Ship to Language Models

Chan Meng
preprint en

Abstract

Software increasingly ships descriptions of itself intended for a language model rather than a person: specifications, constrained-decoding grammars, JSON schemas, vendored context files, and the identity a server states to its host at handshake time. These artifacts are neither compiled nor executed against the thing they describe. The gate deployed over them, almost universally, is reflexive: regenerate the artifact and fail on a difference from the committed copy. We observe that such a gate establishes determinism and cannot, even in principle, establish truth — and that it is nonetheless sound wherever it is deployed, because the artifacts it guards are mechanically derived from their subject, and a derived artifact can be stale but never wrong. That precondition is never stated and never checked. When a generator retypes a fact instead of deriving it, the same gate is vacuous, and it looks identical in continuous integration. We measure what this costs. A census of all 24,131 servers listed on the Model Context Protocol registry, covering the 7,272 unique npm packages among them, finds that of the 5,052 packages whose self-reported identity can be resolved statically, 89.6% retype it rather than derive it, and 41.1% state a version that once tracked the package and has since been left behind — corroborated against each package's own release history, with independently versioned cases excluded. We add six instances across four independent third-party compilers, four of them verified by execution, and a depth study of one production language whose four self-descriptions were all wrong on first check. We claim none of the underlying concepts, all of which are established; we report the precondition, the artifact class, and the measurement. The subject system was substantially co-authored with language models, and we discuss what that implies for which guards matter.

Zenodo (CERN European Organization for Nuclear Research)
Software Engineering Research
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.