Interpreting Character-Level Transformers for Addition and Multiplication Through Integrated Gradients

Transformer models have demonstrated strong performance across a variety of reasoning tasks. However, understanding how these models perform arithmetic reasoning remains an open challenge. This study focuses on investigating the factors that influence arithmetic performance and the interpretability of character-level transformer models. These models were trained on increasingly complex tasks including one-digit arithmetic, two-digit arithmetic, and natural language prompts containing arithmetic expressions embedded within them. Model performance was evaluated by examining the effects of altering training iterations, dataset size, and model configurations. In order to account for training variability, representative two-digit configurations were trained using five independent random seeds and evaluated using a standardized prompt-level procedure. In order to perform the interpretability analysis, Integrated Gradients was utilized to calculate token-level attribution for each input token across zero embedding and mean embedding baselines. The results show that model performance generally improved as training iterations, dataset size, and model configurations increased. Additionally, expanding the character vocabulary allowed the model to process natural language prompts containing arithmetic expressions while generally assigning lower attribution scores to contextual letters and whitespace than for arithmetic tokens. The findings show that the character-level transformer models achieved high accuracy on arithmetic expressions embedded within natural language, while attribution scores were generally higher for arithmetic tokens than for contextual letters and whitespace. This study examines factors associated with the performance of character-level transformer models and provides interpretability evidence of the internal features that contribute to model predictions.