Thinking Harder Doesn't Help AI Catch Silent Failures: Reasoning Effort and Silently Failed Saves in Tool-Using Customer Service Agents
AI agents often report a save as complete when it never happened. Higher reasoning effort is known to make models check their work more often (Sun et al., 2026), so it might catch these failures. This study tested that on 20 airline customer service tasks from τ-bench. In every attempt, the first save returned a normal success response without changing the database. Claude Sonnet 5.5, GPT-6.1 Sol and DeepSeek V4.1 Flash each ran every task at low and high reasoning effort, for 120 attempts; the trap fired in 118 of them. High effort did not catch more failures: 11 of 59 attempts at high effort versus 10 of 59 at low (19% vs 17%). No model's difference was statistically significant. High effort raised reasoning tokens about 2× for Sonnet and 43× for GPT; for GPT it also raised cost 1.5× and customer waiting time from 33 to 58 seconds. No agent looked a booking up again to confirm a save before moving on. All 21 catches happened when the agent later changed the same booking and that save's response showed the earlier change missing; without such a later save, 0 of 90 attempts caught the failure. When numbers stopped adding up, agents usually explained them away, and in 4 of 21 catches they gave the customer an invented cause. Whether a failure was caught depended on the task, not on effort or model.
Authors
- BriAnna Pischke (ORCID: https://orcid.org/0009-0003-6052-9387)
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-11-02
- DOI
- https://doi.org/10.5281/zenodo.23895164
- Citations
- 1
- Primary Topic
- Artificial Intelligence Applications
- Type
- preprint