Does Tool Output Truncation Cause LLM Agents to Answer Incorrectly? A Controlled Experiment with 1,198 LLM Calls

Question. arXiv:2610.36114 ("Context Is a Pipeline, Not a Window") measured that one third of tool results are truncated before reaching the model, and its authors left the causal question open: "Whether the data cut truly causes the accuracy drop is still an open question." Answer. This report answers that question with a controlled experiment over 1,198 real LLM calls, across two models (qwen3.6-flash and nvidia/nemotron-3-nano-omni-30b-a3b-reasoning) and two corpora (one self-authored, one public: DatNguyenAIML/ViLegalExperts on HuggingFace). Design. Oracle setup: the chunk containing the answer is always placed at position [1] of the context, so the only variable that changes is truncation length. Questions are cloze items that quote the source line with the target value masked as [X]. Results. Self-authored corpus, qwen3.6-flash (861 calls): 99.4% correct when the answer is present versus 23.3% when truncated. Gap 76.1 percentage points, z = 23.42, p < 1e-100. Cross-model replication, nemotron-30b (904 calls): 55.3% versus 7.3%. Gap 48.0 pp, McNemar 81 down / 4 up, p = 1.10e-19. Public corpus ViLegalExperts, nemotron-30b (1,198 calls): 67.7% versus 2.3%. Gap 65.4 pp. Confound check: of 87 correct answers despite truncation, 68 (78%) were answer leakage from another retrieved chunk. Removing the confound, true accuracy under missing information is 5.1% (19/373). Behaviour: when the answer is cut, models refuse in 67.3% of cases at a 150-character limit but only 25.0% at 1,200 characters; the remainder fabricate a value (37.8% overall). Framework audit: OpenAI Agents SDK, LangChain and LlamaIndex do not truncate tool output by default. Conclusion. Tool output truncation causes incorrect answers. Measured damage is 48 to 76 percentage points and scales with model capability: stronger models are damaged more. Recommendation. If truncation is unavoidable, cut from the beginning and preserve the tail (answers positioned late in the chunk survived at 40.4% versus 0.0% for answers at the start). Better: do not truncate, or use a recoverable compression scheme. Includes both an English report and a Vietnamese report. All data, code and results are retained; scripts are named in the appendix.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-19
DOI
https://doi.org/10.5281/zenodo.24376531
Primary Topic
Topic Modeling
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

Does Tool Output Truncation Cause LLM Agents to Answer Incorrectly? A Controlled Experiment with 1,198 LLM Calls

7EN Labs
Zenodo (CERN European Organization for Nuclear Research)
Topic Modeling
article

Does Tool Output Truncation Cause LLM Agents to Answer Incorrectly? A Controlled Experiment with 1,198 LLM Calls

7EN Labs
article en

Abstract

Question. arXiv:2610.36114 ("Context Is a Pipeline, Not a Window") measured that one third of tool results are truncated before reaching the model, and its authors left the causal question open: "Whether the data cut truly causes the accuracy drop is still an open question." Answer. This report answers that question with a controlled experiment over 1,198 real LLM calls, across two models (qwen3.6-flash and nvidia/nemotron-3-nano-omni-30b-a3b-reasoning) and two corpora (one self-authored, one public: DatNguyenAIML/ViLegalExperts on HuggingFace). Design. Oracle setup: the chunk containing the answer is always placed at position [1] of the context, so the only variable that changes is truncation length. Questions are cloze items that quote the source line with the target value masked as [X]. Results. Self-authored corpus, qwen3.6-flash (861 calls): 99.4% correct when the answer is present versus 23.3% when truncated. Gap 76.1 percentage points, z = 23.42, p < 1e-100. Cross-model replication, nemotron-30b (904 calls): 55.3% versus 7.3%. Gap 48.0 pp, McNemar 81 down / 4 up, p = 1.10e-19. Public corpus ViLegalExperts, nemotron-30b (1,198 calls): 67.7% versus 2.3%. Gap 65.4 pp. Confound check: of 87 correct answers despite truncation, 68 (78%) were answer leakage from another retrieved chunk. Removing the confound, true accuracy under missing information is 5.1% (19/373). Behaviour: when the answer is cut, models refuse in 67.3% of cases at a 150-character limit but only 25.0% at 1,200 characters; the remainder fabricate a value (37.8% overall). Framework audit: OpenAI Agents SDK, LangChain and LlamaIndex do not truncate tool output by default. Conclusion. Tool output truncation causes incorrect answers. Measured damage is 48 to 76 percentage points and scales with model capability: stronger models are damaged more. Recommendation. If truncation is unavoidable, cut from the beginning and preserve the tail (answers positioned late in the chunk survived at 40.4% versus 0.0% for answers at the start). Better: do not truncate, or use a recoverable compression scheme. Includes both an English report and a Vietnamese report. All data, code and results are retained; scripts are named in the appendix.

Zenodo (CERN European Organization for Nuclear Research)
Topic Modeling
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon