Frequency Domain Adaptive Filters in Vision Transformers for Small‐Scale Datasets

ABSTRACT Transformers have achieved remarkable success in computer vision, but their reliance on self‐attention mechanisms poses challenges for small‐scale datasets due to high computational demands and data requirements. This paper introduces the Multi‐Head Adaptive Filter Frequency Vision Transformer (MAF‐FViT), a Vision Transformer model that replaces self‐attention with frequency‐domain adaptive filters. MAF‐FViT leverages multi‐head adaptive filtering in the frequency domain to capture essential features with reduced computational complexity, providing an efficient alternative for vision tasks on limited data. Training is carried out from scratch without the need for pretraining on large‐scale datasets. The proposed MAF‐FViT model demonstrates strong performance on various image classification tasks, achieving competitive accuracy with a lower parameter count and faster processing times compared to self‐attention‐based models and other models employing alternative token mixers. The multi‐head adaptive filters enable the model to capture complex image features effectively, preserving high classification accuracy while minimising computational load. The results demonstrate that frequency‐domain adaptive filters offer an effective alternative to self‐attention, enabling competitive performance on small‐scale datasets while reducing training time and memory requirements. MAF‐FViT opens avenues for resource‐efficient transformer models in vision applications, making it a promising solution for settings constrained by data or computational resources.