Contextual priority attention enables linear time sequence modeling in transformers

The quadratic computational complexity of the self-attention mechanism in Transformer models severely constrains their applicability to long sequence inputs. We propose Contextual Priority Attention (CPA), a novel attention mechanism that reduces computational complexity from [Formula: see text] to [Formula: see text] theoretically, with experimentally observed linear scaling, while preserving the essential contextual modeling capabilities of traditional attention. Unlike existing efficient attention variants that approximate the full attention matrix, CPA fundamentally reimagines attention as a global-context-driven priority system. Our approach first computes a Global Context Vector (GCV) to capture the overall sequence information, then uses this context to assign priority scores to individual tokens, enabling sparse attention allocation. In encoder-decoder architectures, we apply CPA to the encoder while maintaining standard attention in the decoder, achieving substantial efficiency improvements while preserving full model expressivity for sequence-to-sequence tasks. Through theoretical analysis and extensive experimentation across language understanding, translation, and long document tasks, we demonstrate that CPA achieves comparable performance to standard self-attention while offering significant efficiency gains. On long-sequence benchmarks (8K+ tokens), CPA outperforms traditional Transformers and state-of-the-art efficient attention variants while using substantially fewer computational resources. Our findings suggest that pairwise token interactions may be unnecessary for effective contextual modeling, opening new avenues for efficient large-scale language models.