Exploiting representational redundancy in transformer hidden states through token representation reuse

Abstract Transformer-based large language models construct rich contextualized token representations across multiple layers, yet redundancy within their intermediate hidden-state spaces remains insufficiently explored as an inference-time representation property. This paper introduces token representation reuse (TRR), an inference-time framework that identifies highly similar contextual token representations and replaces selected hidden states with preceding anchor representations under similarity and reuse-budget constraints. Unlike token pruning or merging, TRR preserves the original sequence length and positional structure and requires neither model retraining nor parameter modification. We further provide an end-to-end complexity analysis of representation discovery and characterize the local conditions under which identical hidden states can, in principle, permit projection reuse, while explicitly showing that such equality does not generally persist across subsequent transformer layers. Experiments on BoolQ, AGNews, SST-2, and TruthfulQA using Qwen2.5-1.5B-Instruct and Llama-3-8B demonstrate representation reuse ratios of up to 12.40% while largely preserving downstream predictive performance. Under the frozen default configuration, paired statistical analysis shows no significant accuracy change on AGNews or SST-2, while TruthfulQA exhibits a statistically significant improvement of $$+2.92$$ + 2.92 percentage points on Qwen2.5-1.5B-Instruct (95% CI $$[+1.15,+4.69],$$ [ + 1.15 , + 4.69 ] , p = 0.001); BoolQ shows a significant decrease of $$-1.50$$ - 1.50 percentage points. End-to-end runtime profiling of the same TRR algorithmic procedure shows that the current dense implementation incurs additional latency because the $$ O(n^2d) $$ O ( n 2 d ) similarity-discovery cost is not offset by hardware-level computation skipping. Within the evaluated models, tasks, and sequence lengths, these results demonstrate the feasibility of controlled hidden-state representation reuse while identifying efficient similarity discovery and low-level computational reuse as necessary directions for investigating practical scalability and future acceleration.