Integrating Spatial Information into Global Context: Summary Vision Transformer (S-ViT)

Vision Transformers (ViTs), inspired by transformers originally developed for Natural Language Processing (NLP), have demonstrated their capability to outperform Convolutional Neural Networks (CNNs) through their self-attention mechanism. Despite their advantages, the training of ViTs faces challenges with smaller datasets, primarily due to their lack of inherent inductive biases, a key strength of traditional CNNs. To address this shortcoming, this study introduces a novel plug-and-play method that integrates spatial, hierarchical, and locality information from CNNs into ViTs. We employ ResNet-18 as a feature extractor to enrich the ViT architecture, creating the Summary Vision Transformer (S-ViT). The performance of S-ViT was evaluated across multiple public datasets, including CIFAR-10, CIFAR-100, Pets-37, and Flowers-102, under various training conditions—both from scratch and in transfer learning scenarios. Further, we also compare S-ViT with other plug-and-play methods. Our findings reveal that S-ViT not only surpasses baseline ViT models in terms of accuracy but also demonstrates enhanced flexibility and scalability across different architectures.