Patch-Range Transformer (PRT): Efficient Local Attention for Vision Transformers

Patch-Range Transformer (PRT): Efficient Local Attention for Vision Transformers Vision Transformers (ViT) have emerged as a powerful architecture for image classification by leveraging global self-attention mechanisms. However, their reliance on full pairwise attention results in quadratic computational complexity with respect to the number of image patches and limits their ability to effectively capture local spatial structures. This leads to increased computational cost and reduced data efficiency, particularly when training on small to medium-scale datasets. In this work, we propose the Patch-Range Transformer (PRT), a simple yet effective modification to the standard Vision Transformer that introduces spatially constrained local attention without increasing model complexity. Specifically, each patch token attends only to its neighboring patches within a fixed Chebyshev distance on a two-dimensional patch grid. Meanwhile, a global classification (CLS) token maintains full attention across all patches, enabling efficient global information aggregation and preserving long-range dependencies. Unlike prior approaches such as Swin Transformer, which rely on hierarchical architectures, shifted window mechanisms, and additional parameters, PRT achieves efficient local attention through a lightweight masking strategy. This design introduces no additional parameters, requires minimal architectural changes, and can be seamlessly integrated into existing ViT frameworks. The key contributions of this work are as follows. First, we introduce a Patch-Range attention mechanism based on Chebyshev distance for efficient spatially local interactions. Second, we propose a CLS-based global communication strategy that enables cross-region information flow without complex architectural modifications. Third, we demonstrate a parameter-efficient design that maintains identical model size while reducing attention FLOPs by up to 61%. Finally, we provide a comprehensive ablation study showing that a moderate attention range yields the optimal balance between accuracy and computational efficiency. Extensive experiments on benchmark datasets, including CIFAR-10, CIFAR-100, and Caltech-101, are conducted under identical training conditions without pretrained weights. The results show that PRT consistently outperforms the standard ViT while maintaining the same parameter count. On CIFAR-10, PRT achieves 82.11% accuracy, improving over ViT by 1.00% while reducing FLOPs by 61%. On CIFAR-100, it surpasses both ViT and Swin Transformer while using approximately ten times fewer parameters than Swin. On Caltech-101, PRT achieves significant accuracy gains over both baselines. An ablation study on the attention range parameter further reveals that R = 2 achieves the best trade-off between computational efficiency and classification performance, attending to only a subset of patches while maintaining optimal accuracy. These findings highlight the effectiveness of structured local attention in improving efficiency without sacrificing model performance. Overall, the Patch-Range Transformer provides a scalable, parameter-efficient, and computationally optimized alternative to global attention mechanisms in Vision Transformers. Its simplicity, effectiveness, and strong empirical performance make it well-suited for real-world applications and resource-constrained environments. Code and resources:GitHub: https://github.com/Balamurugan245/Patch-Range-TransformerKaggle: https://www.kaggle.com/code/balamuruganaiml/brain-tumor-mri-prt