RevOrder: A novel equation format for arithmetic operations in language models

Abstract This paper proposes to understand arithmetic operations in Language Models (LM) by framing them as digit‐based reasoning challenges. Our research focuses on arithmetic optimization challenges specific to LLMs, not on solving mathematical word problems. We introduce a metric called the Count of Sequential Intermediate Digits (CSID), which measures the complexity of arithmetic equations by counting the missing steps in digit reasoning. Our empirical findings suggest that increasing the model size does little to improve the handling of equations with high CSID values. We propose RevOrder, a method that incorporates techniques such as reversing the output order, step‐by‐step decomposition, and rollback mechanisms to maintain a low CSID, thereby enhancing the solvability of arithmetic equations in LMs. RevOrder also introduces a more compact reasoning process, which reduces the token requirements without affecting the CSID, significantly enhancing token efficiency. Comprehensive testing shows that RevOrder achieves perfect accuracy in operations such as addition, subtraction, and multiplication, and substantially improves performance in division tasks, especially with large numbers where traditional models falter.