Deterministic, Reproducible, and Wrong: How Our Own Scorer Manufactured an Agent-Memory Finding, and the Guard That Would Have Caught It
In the agent-memory literature we surveyed, provenance is primarily used to make agents trust retrieved context less: poisoning defenses, injection gating, watermarking, and transfer integrity all point in that direction. This paper asks the inverse question. When a harness delivers a legitimate, correct note into an agent's context, and the note asserts something the agent can neither corroborate from its workspace nor arrive at through its default behavior, does attaching a corroboration affordance, a way for the agent to check the claim for itself, change what the agent does where authoritative wording does not? We report a pilot that does not answer that question, and the reason transfers further than the answer would have. Our first pass produced a clean headline: delivery of the note was confirmed from two separate sides, and the agent still committed the exact mistake the note existed to prevent. That headline was an artifact of our own scorer, a deterministic program with no language model anywhere in it. Its mistake signature matched shell text anywhere in a command, so it fired on a read, cat deploy.sh. The transcript instead shows the agent declining to run the script and citing the delivered note as its reason. Re-scoring every preserved session in the reported grid at zero additional model cost cleared every flagged leg. The pilot could not answer its question, but it exposed an evaluation failure whose diagnosis, correction, and guard we consider the primary contribution of this work. What survives is smaller and, we think, more useful: an outcome-level negative result for the provenance-only note variant; one fully documented specimen of a delivered note entering an agent's stated reasoning, with the counterfactual left open; a shell matcher that fires only at execution position; and a machine-readable contradiction guard between what the agent did and what the workspace shows, which fails a record automatically when they disagree. The wrong number was never published. Catching it was ordinary quality control before submission; reporting it is not. We report a direction, a negative result, and a list of the things our own harness got wrong.
Authors
- Swapnanil Saha
Publication Details
- Journal
- arXiv (Cornell University)
- Published
- 2026-08-24
- DOI
- https://doi.org/10.5281/zenodo.22079951
- Primary Topic
- Personal Information Management and User Behavior
- Type
- preprint