SAG-ViT: a scale-aware, high-fidelity patching approach with graph attention for vision transformers

Abstract Vision Transformers (ViTs) have redefined image classification by leveraging self-attention to capture complex patterns and long-range dependencies between image patches. However, a key challenge for ViTs is efficiently incorporating multi-scale feature representations, which is inherent in convolutional neural networks (CNNs) through their hierarchical structure. Graph transformers have made strides in addressing this by leveraging graph-based modeling, but they often lose or insufficiently represent spatial hierarchies, especially since redundant or less relevant areas dilute the image’s contextual representation. To bridge this gap, we propose SAG-ViT, a Scale-Aware Graph Attention ViT that integrates the multi-scale feature capabilities of CNNs, the representational power of ViTs, and graph-attended patching to enable richer contextual representation. Using EfficientNetV2 as a backbone, the model extracts multi-scale feature maps, dividing them into patches to preserve richer semantic information compared to directly patching the input images. The patches are structured into a graph using spatial and feature similarities, where a Graph Attention Network (GAT) refines the node embeddings. This refined graph representation is then processed by a Transformer encoder, capturing long-range dependencies and complex interactions. Evaluated on six diverse image classification benchmarks, SAG-ViT achieves an F1 score of 0.9574 on CIFAR-10 and 0.9958 on GTSRB, to validate its consistent improvements across multiple backbones. Our code and weights are available at https://github.com/shravan-18/SAG-ViT .