Precise Records, Unstable Meanings: Measurement Validity and Unsupported Claims Derived from AI Agent Telemetry

Overview Precise Records, Unstable Meanings is a measurement-validity audit of AI agent telemetry. It asks a practical question: when systems record transcripts, ledger events, anomaly scores, traces, and timestamps, what do those records actually support us in claiming about sessions, tasks, outcomes, behavior, and failure? Study The preprint reports a naturalistic audit of one coding-agent orchestration-harness deployment observed over twelve weeks. The frozen corpus comprised 41,495 transcript files and six auxiliary telemetry streams. Each candidate claim was checked for producer provenance, analytical unit, construct, population, observation window, denominator, validation evidence, and sensitivity to alternative treatments of inter-event gaps. Findings Valid record-level quantities did not support several broader construct-level interpretations. Transcript-file spans did not establish logical-session or task duration; 46 of 87 files spanning more than eight hours fell below that threshold after removal of their largest internal gap. Of 28,055 claims-ledger rows, 27,938 were automatic retrieval-reflex events, while the remaining 117 heterogeneous entries did not establish completed tasks or correct outcomes. An anomaly instrument flagged approximately 65% of scored turns, but without independent behavioral labels this was an instrument-positive rate, not an estimate of failure prevalence. Telemetry-to-Claim Gate The paper introduces a practical gate for documenting producer, unit, construct, population, window, denominator, validation, and sensitivity before telemetry is used in agent evaluation, observability, governance, or system redesign. Relevance This work contributes to AI agent evaluation, agent observability, coding-agent telemetry, trajectory and trace analysis, construct validity, measurement validity, telemetry provenance, audit trails, and the study of unsupported claims derived from operational records. Related work Conceptual companion: The Generative Horizon: Applied Hermeneutics, Linguistic Attractors, and the Limits of Model Self-Report, doi:10.5281/zenodo.21659634. The empirical study does not depend on, test, or validate that conceptual framework.