Who Validates the Validator? Instrument failure in the shape of the hypothesis — a pre-registered measurement programme with a public correction ledger

# v6 opening block — EN (approved 2026-08-08, ships with the next substantive version) What this is. A pre-registered measurement of whether multi-agent LLM systems faithfully report the tool calls they claim to have made — together with the validated audit instrument used to take it, and the full correction history of that instrument. Every tool call passes through a logging proxy the model can neither see nor write; the log is the primary datum, and a deterministic scorer compares reported execution against actual execution. The headline result, and its scope. Across 1,250 runs (30 sealed tasks × {2, 5, 10, 20} agents, one pinned open-weights engine), 10/1250 = 0.80% of claimed tool dispatches had no backing execution. This is an estimate for prose-reporting regimes, where agents narrate their tool use in text; under native structured tool calling, the call is the claim and cannot diverge. A defective earlier version of the scorer would have confirmed the pre-registered scaling hypothesis at p ≈ 10⁻⁶; the corrected instrument, over the same intact data, did not. That difference — and the thirty numbered findings behind it — is what this record documents. Its first preregistered follow-up experiment (720 runs, two engines, OSF ud9c4) found no evidence that the error representation changes silent substitution — and a 3× engine-stack-associated difference that dwarfed it. What you can use it for. Auditing an agentic system of your own: the receipt pattern and the scorer are directly reusable, and the reference implementation ships a CI-ready gate (PASS / FAIL / INCONCLUSIVE exit codes). Validating an evaluator of your own: the fault-injection matrix, the negative-control classes and the mutation protocol apply to any scoring instrument, not just this one. Or as a worked template for pre-registered measurement with a machine-checked correction record. The software. Reference implementation: https://github.com/glasseymour/dispatch-fidelity (Apache-2.0, no runtime dependencies, `dispatch-audit` CLI). Versioned, checksummed software releases are archived under the software concept DOIs (manual v0.3.0 series: 10.5281/zenodo.21841082; automatic v0.3.1+ series: 10.5281/zenodo.21850624). The two records answer different questions — this one documents what was measured and how the instrument was validated; the software record identifies which bytes you ran — and reproducing a result needs both. # v6 nyitóblokk — HU iker (jóváhagyva 2026-08-08) Mi ez. Előregisztrált mérés arról, hogy a több-agentes LLM-rendszerek hűen jelentik-e az általuk állított eszközhívásokat — együtt a mérést végző, validált audit-műszerrel és a műszer teljes korrekciós történetével. Minden eszközhívás egy naplózó proxyn halad át, amelyet a modell se nem lát, se nem írhat; az elsődleges adat a napló, és egy determinisztikus pontozó veti össze a jelentett végrehajtást a ténylegessel. A fő eredmény és a hatóköre. 1250 futáson át (30 lepecsételt feladat × {2, 5, 10, 20} agent, egyetlen rögzített nyílt-súlyú motor) az állított eszköz-dispatchek 10/1250 = 0,80%-a mögött nem állt végrehajtás. Ez a próza-jelentéses rezsimek becslése, ahol az agentek szövegben számolnak be az eszközhasználatról; natív strukturált eszközhívásnál a hívás maga az állítás, és nem térhet el. A pontozó egy korábbi, hibás változata p ≈ 10⁻⁶ mellett megerősítette volna az előregisztrált skálázási hipotézist; a javított műszer ugyanazon a sértetlen adaton nem. Ez a különbség — és a mögötte álló harminc számozott lelet — az, amit ez a rekord dokumentál. Az első előregisztrált követő kísérlet (720 futás, két motor, OSF ud9c4) nem talált bizonyítékot arra, hogy a hiba-reprezentáció megváltoztatná a néma helyettesítést — és talált egy közel háromszoros, engine-stackhez társuló különbséget, amely mellett a hibafelület-kontraszt eltörpült. Mire használható. Saját agentikus rendszer auditjára: a nyugta-minta és a pontozó közvetlenül újrahasznosítható, a referencia-implementáció CI-kész kaput ad (PASS / FAIL / INCONCLUSIVE kilépési kódok). Saját értékelő validálására: a hibainjekciós mátrix, a negatív kontroll-osztályok és a mutációs protokoll bármely pontozó műszerre alkalmazható, nem csak erre. Vagy kidolgozott sablonként előregisztrált méréshez, gépileg ellenőrzött korrekciós rekorddal. A szoftver. Referencia-implementáció: https://github.com/glasseymour/dispatch-fidelity (Apache-2.0, futásidejű függőség nélkül, `dispatch-audit` CLI). A verziózott, checksumolt szoftver-kiadásokat a szoftver concept DOI-k archiválják (kézi v0.3.0 sorozat: 10.5281/zenodo.21841082; automatikus v0.3.1+ sorozat: 10.5281/zenodo.21850624). A két rekord különböző kérdésre válaszol — ez azt dokumentálja, mit mértünk és hogyan validáltuk a műszert; a szoftver-rekord azt azonosítja, melyik byte-okat futtattad —, és egy eredmény reprodukálásához mindkettő kell. v6 (2026-08-13). This version carries the programme's first preregistered follow-up experiment: 720 runs, two weight-sealed local engines, one injected recoverable tool failure per run (OSF ud9c4, registered before data collection). Result: no evidence that the error representation (normalized type vs native message) changes silent substitution — paired-difference intervals exclude large effects — alongside a 3× engine-stack-associated descriptive difference (the design does not separate model, architecture and serving stack) and a reversed tool-substitution profile, alongside a ~30% inconclusive burden that is itself reported as a finding about report-contract discipline. All deviations, the regime matrix, the full run corpus and a one-command re-derivation ship in this record (v6_error_surface_package.zip). The correction ledger grows by one instrument-boundary finding (#31), found by this experiment's own harness. v5 (2026-08-08) — record integrity, power boundary, regime surfaces, finding register. Correction #30: the showcase builder wrote platform-default line endings while claiming determinism; output is now pinned to LF, and a new RECORD_MANIFEST.sha256 binds every file of this record — the showcase layer previously sat outside every gate. A Linux rebuild is now byte-identical to the Windows-built uploads on all eight showcase files, measured rather than assumed. Power boundary: the pre-registered design was sized for a base rate near 0.15; at the observed 0.80% it has no meaningful power for H2 — the null result is absence of evidence, not evidence of absence, now stated beside the number in both results documents. Measurement regimes: the report contract (fenced-json-results/1), protocol topology (orchestrator-worker-text-protocol/1) and tool-name surface (fixed-toolset-disclosed-names/1) join the error surface with named regime identifiers in docs/measurement_regime.json: the 0.80% is an estimate for prose-reporting regimes, not for agentic systems in general — under native structured tool calling the call is the claim and cannot diverge. Finding register: all thirty findings with date, channel, layer and phase in docs/finding_register.json; exposure units are not yet modeled and the recorded proxies are named as proxies. The registered primary outcome is unchanged: 10/1250 = 0.80%. v4 (2026-08-08) — boundary condition on the silent-substitution rate. Correction item #14 reported a silent-substitution rate of 18/91 = 19.8% without stating what the agent could SEE after a tool failed. The logging proxy does not merely observe: it produces the error representation the agent then reasons from — ERROR:<ExceptionType>, with no message, no traceback and no native exception propagated. The proxy therefore functioned as an interface intervention as well as a recorder, and the estimate is conditional on that representation; it should not be compared directly with measurements taken under native exceptions, structured framework error objects or full tracebacks unless error-surface policy is treated as an experimental factor. All 18 cases also arose on a single tool (text_stat). docs/error_surface.json names the regime (proxy-normalized-type-only/1) so a future figure is a different NAMED regime rather than an unexplained disagreement. The registered primary outcome is unaffected: 10/1250 = 0.80% — it asks whether a claimed call happened, which does not depend on how a failure was displayed. The papers are deliberately unchanged: dated texts with shipped PDFs and no reproducible build recipe. v3 (2026-08-07) — packaging-only correction. The parameterless release gate previously referenced the v1 archive name, so after v2 shipped it gated the earlier archive — and the parameterless invocation is the one the reproduction instructions describe as "one command". Default package discovery is now version-independent and fails closed on zero or multiple candidates. The correction protocol gains section 26 for this item, and sections 16–25 pointing to findings recorded in the instrument's portable form (dispatch-fidelity). No research data, task outputs, scores, statistical analyses, registered outcomes or conclusions changed. Gate logs for Windows and Linux verify the same archive hash. v2 (2026-08-07) — external finding #15. This version adds correction item #15 to the correction protocol: cross-run evidence splicing. File-level integrity proves that a file is what it claims to be; it proves nothing about whether two files belong together. Genuine, hash-valid artifacts drawn from different runs assemble into a package that every v1 check passes. The binding material was already in the package and had never been checked: the run manifest commits sha256(nonce) before execution, and the plaintext nonce is recoverable only from a genuine canary receipt in the tool log. Two new analyses run that check — analysis/check_run_binding.py (B1–B5; B3 proven for 125 of 129 run groups, four declared unprovable in docs/binding_unprovable.txt) and analysis/x2_splice.py (X2 injection class; manifest and tool-log splices detected at 100.000%, specificity 120/120). T