Does Tool Output Truncation Cause LLM Agents to Answer Incorrectly? A Controlled Experiment with 1,198 LLM Calls
Question. arXiv:2610.36114 ("Context Is a Pipeline, Not a Window") measured that one third of tool results are truncated before reaching the model, and its authors left the causal question open: "Whether the data cut truly causes the accuracy drop is still an open question." Answer. This report answers that question with a controlled experiment over 1,198 real LLM calls, across two models (qwen3.6-flash and nvidia/nemotron-3-nano-omni-30b-a3b-reasoning) and two corpora (one self-authored, one public: DatNguyenAIML/ViLegalExperts on HuggingFace). Design. Oracle setup: the chunk containing the answer is always placed at position [1] of the context, so the only variable that changes is truncation length. Questions are cloze items that quote the source line with the target value masked as [X]. Results. Self-authored corpus, qwen3.6-flash (861 calls): 99.4% correct when the answer is present versus 23.3% when truncated. Gap 76.1 percentage points, z = 23.42, p < 1e-100. Cross-model replication, nemotron-30b (904 calls): 55.3% versus 7.3%. Gap 48.0 pp, McNemar 81 down / 4 up, p = 1.10e-19. Public corpus ViLegalExperts, nemotron-30b (1,198 calls): 67.7% versus 2.3%. Gap 65.4 pp. Confound check: of 87 correct answers despite truncation, 68 (78%) were answer leakage from another retrieved chunk. Removing the confound, true accuracy under missing information is 5.1% (19/373). Behaviour: when the answer is cut, models refuse in 67.3% of cases at a 150-character limit but only 25.0% at 1,200 characters; the remainder fabricate a value (37.8% overall). Framework audit: OpenAI Agents SDK, LangChain and LlamaIndex do not truncate tool output by default. Conclusion. Tool output truncation causes incorrect answers. Measured damage is 48 to 76 percentage points and scales with model capability: stronger models are damaged more. Recommendation. If truncation is unavoidable, cut from the beginning and preserve the tail (answers positioned late in the chunk survived at 40.4% versus 0.0% for answers at the start). Better: do not truncate, or use a recoverable compression scheme. Includes both an English report and a Vietnamese report. All data, code and results are retained; scripts are named in the appendix.
Authors
- 7EN Labs
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-11-19
- DOI
- https://doi.org/10.5281/zenodo.24376531
- Primary Topic
- Topic Modeling
- Type
- article
- Field-Weighted Citation Impact
- 0.00