Treat Every Claim as a Hypothesis: Measurements from a Multi-Model Agent Fleet

A field report from operating a small fleet of LLM agents from several model families on real engineering and research work. Negative results first. Topics: a command-classification gate for a code-executing agent and the bypass classes that review found (and that a later revision left open); measurements in an agent loop, including a benchmark that leaked its answers through comments the agents could read, and an episode in which a model wrote an autouse conftest.py so that a flaky test would pass; cross-family blind evaluation of a document pipeline, including an injected OCR fault that was detected and ignored, a repaired version that passed the known variant, and a first blind variant (commit-reveal) that failed because the pseudonymisation layer removed a negation; and a verified case base of agent claims contradicted by artifacts, in which the most frequent source is the orchestrator itself. All case data used in the evaluations is synthetic. Section 7 states what the report does not claim. Version 2: typeset in LaTeX and adds figures (system and gate diagram, review timeline, evaluation chart, document pipeline with the fault-injection and commit-reveal points, case-base chart) and fixes a list-formatting error in Section 3.3 of version 1. No numbers were changed; every number in the figures appears in the text.