Study on Hybrid Models Combining Residual Networks and Attention/Vision Transformer Models

This paper proposes two hybrid models, MH-ResNet and ResViT, that integrate the strengths of residual networks (ResNet) and vision transformers (ViT) to enhance image classification performance. MH-ResNet incorporates a Multi-Head Self-Attention (MHSA) module in parallel with the residual blocks of ResNet, enabling simultaneous learning of local and global features. ResViT fuses feature vectors extracted from parallel ResNet and ViT branches, leveraging both the inductive biases of CNNs and the global contextual information of transformers. Experimental evaluations on the CIFAR-100 and Tiny ImageNet datasets show that the proposed models outperform conventional CNNs (ResNet and PyramidSepDrop), ViTs, and hybrid models such as ReViT, ResNet_BAM, and ResNet_WCAM. Notably, MH-ResNet based on ResNet18 achieves 89.52% accuracy with only 16M parameters, proving its suitability for lightweight inference scenarios. Conversely, the ResNet18-based ResViT with transfer learning achieves 94.01% accuracy but requires 99.36M parameters, highlighting a trade-off between accuracy and model complexity.