Thinking Harder Doesn't Help AI Catch Silent Failures: Reasoning Effort and Silently Failed Saves in Tool-Using Customer Service Agents

AI agents often report a save as complete when it never happened. Higher reasoning effort is known to make models check their work more often (Sun et al., 2026), so it might catch these failures. This study tested that on 20 airline customer service tasks from τ-bench. In every attempt, the first save returned a normal success response without changing the database. Claude Sonnet 5.5, GPT-6.1 Sol and DeepSeek V4.1 Flash each ran every task at low and high reasoning effort, for 120 attempts; the trap fired in 118 of them. High effort did not catch more failures: 11 of 59 attempts at high effort versus 10 of 59 at low (19% vs 17%). No model's difference was statistically significant. High effort raised reasoning tokens about 2× for Sonnet and 43× for GPT; for GPT it also raised cost 1.5× and customer waiting time from 33 to 58 seconds. No agent looked a booking up again to confirm a save before moving on. All 21 catches happened when the agent later changed the same booking and that save's response showed the earlier change missing; without such a later save, 0 of 90 attempts caught the failure. When numbers stopped adding up, agents usually explained them away, and in 4 of 21 catches they gave the customer an invented cause. Whether a failure was caught depended on the task, not on effort or model.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-02
DOI
https://doi.org/10.5281/zenodo.23895164
Citations
1
Primary Topic
Artificial Intelligence Applications
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
preprint

Thinking Harder Doesn't Help AI Catch Silent Failures: Reasoning Effort and Silently Failed Saves in Tool-Using Customer Service Agents

BriAnna Pischke
1 citations
Zenodo (CERN European Organization for Nuclear Research)
Artificial Intelligence Applications
preprint

Thinking Harder Doesn't Help AI Catch Silent Failures: Reasoning Effort and Silently Failed Saves in Tool-Using Customer Service Agents

BriAnna Pischke
preprint en
1 citations

Abstract

AI agents often report a save as complete when it never happened. Higher reasoning effort is known to make models check their work more often (Sun et al., 2026), so it might catch these failures. This study tested that on 20 airline customer service tasks from τ-bench. In every attempt, the first save returned a normal success response without changing the database. Claude Sonnet 5.5, GPT-6.1 Sol and DeepSeek V4.1 Flash each ran every task at low and high reasoning effort, for 120 attempts; the trap fired in 118 of them. High effort did not catch more failures: 11 of 59 attempts at high effort versus 10 of 59 at low (19% vs 17%). No model's difference was statistically significant. High effort raised reasoning tokens about 2× for Sonnet and 43× for GPT; for GPT it also raised cost 1.5× and customer waiting time from 33 to 58 seconds. No agent looked a booking up again to confirm a save before moving on. All 21 catches happened when the agent later changed the same booking and that save's response showed the earlier change missing; without such a later save, 0 of 90 attempts caught the failure. When numbers stopped adding up, agents usually explained them away, and in 4 of 21 catches they gave the customer an invented cause. Whether a failure was caught depended on the task, not on effort or model.

Zenodo (CERN European Organization for Nuclear Research)
Artificial Intelligence Applications
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon
Thinking Harder Doesn't Help AI Catch Silent Failures: Reasoning Effort and Silently Failed Saves in Tool-Using Customer Service Agents — BriAnna Pischke · Zenodo (CERN European Organization for Nuclear Research) (2026) | TGRS Research Map | TGRS