Reconciling Divergent Views Through a Critical Analysis of Iterative Self-Improvement in LLMs

Several attempts have aimed to extend reasoning in Large Language Models (LLMs) via iterative self-improvement. This has given rise to two camps of papers: papers like “Reflexion: Language Agents with Verbal Reinforcement Learning” (NeurIPS 2023) and “Reason for Future, Act for Now: A Principled Framework for Autonomous LLM Agents with Provable Sample Efficiency” (ICML 2024) that support iterative reasoning to improve problem-solving, and papers like “Large Language Models Cannot Self-Correct Reasoning Yet” (ICLR 2024) and “On the Self-Verification Limitations of Large Language Models on Reasoning and Planning Tasks” (ICLR 2025) that show that multiple feedback loops may hurt rather than help LLMs on reasoning tasks.<br/><br/>In this work, we investigate the sources of errors that arise in the iterative reasoning pipeline, in an effort to reconcile these two camps of thought. We decompose the pipeline into the interpretable stages of Generation, Verification, Reflection, and Recalibration, and perform a cross-method analysis to understand the sources of different errors. We analyze both methods that show improvements due to iterative reasoning (like Reflexion) and methods that do not (like LLM+LLM Backprompting), using intermediate reasoning steps. We run these on CacheSaver, a high-efficiency caching library, to reduce costs, improve reproducibility, and streamline debugging. We use GPT-4.1 nano as the base LLM.<br/><br/>We find that most tasks fail due to Verification errors. This stage accounts for ~70% of reasoning errors (mismatches with ground truth) - across both methods (Reflexion with the last reasoning trace in context and Backprompting with LLM+LLM) and across three task domains comprising 200 diverse problems: question answering (HotpotQA), sequential decision making (ALFWorld), and Python programming (HumanEval). Reflection performance improves with Verification accuracy, and vice versa. About 5% of samples also fail in the final Generation step, even though the thoughts generated during Reflection are faithful to the answer.<br/><br/>In ReAct agents, requiring external context retrieval, the most common error appears to result from the chosen search query being underspecified. It retrieves the wrong page, and thus, even with further Verification and Reflection, it still finishes with the incorrect label. For sequential decision making, the Generation stage repeatedly proposes the same non-progressing action, and the Reflection stage often has low granularity - e.g., “try different action” rather than a concrete causal fix. Many memories describe being “stuck in a loop,” continually trying to perform the same action (e.g., repeatedly trying to heat an apple without opening the microwave, or to put an item in without opening it).<br/><br/>This suggests several recommendations for iterative reasoning. Verification accuracy benefits all domains, but its impact is most pronounced in programming tasks, where improved test cases significantly enhance reliability. Discouraging re-selection of identical actions in consecutive steps, unless the environment state has changed, also helps. Verifying that final outputs match the reasoning chain can increase confidence when deploying these methods in practice, since correct reasoning chains can sometimes lead to erroneous answers. We used GPT-4.1 nano in this study; in the future we will evaluate additional base models to substantiate the generalizability of our findings.