Exactly Once Across Retries and Restarts: Measuring Duplicate and Stale Side Effects in Tool-Using LLM Agents
When a tool call fails, an agent cannot tell if the action went through. What it reads next, the error or the resume note, often decides whether it repeats a committed step. We hold the committed state of a simulated ledger fixed and change only that text, across 12,733 episodes, six models and four providers. We also count duplicates as a server that honours idempotency keys would see them (post hoc). Replacing a bare "internal server error" with a status saying the outcome is unknown cut duplicates for Haiku 4.5 and Qwen3.6-27B from over 0.6 to at most 0.19 in every wording, with or without keys honoured. For Sonnet 5 and Fable 5 it did no better than a timeout, and honouring keys removes half or more of its effect. Documented API errors that say the outcome is unknown caused fewer duplicates than generic ones. Restarted with only the task, 115/192 sessions redid a committed step with keys honoured (155/192 as run). A two-sentence note that an earlier session was interrupted made every model check the log first, removing nearly all duplicates. On Stripe test mode neither text left duplicates, though rates matched the simulator in under half the cells.
Authors
- Sanjana Gopnal Swamy
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-11-12
- DOI
- https://doi.org/10.5281/zenodo.24165777
- Primary Topic
- Software System Performance and Reliability
- Type
- preprint