AGSAFormer: An Adaptive Ghost-MLP Gate and Single-Head Self-Attention Transformer for Image Classification

With the advent of deep learning, Transformers have been successfully adapted to computer vision tasks, garnering significant attention. While Vision Transformers (ViTs) demonstrate superior performance compared to traditional models, their high computational requirements impede widespread adoption. In this paper, we propose an efficient backbone AGSAFormer, a hybrid convolutional neural network (CNN) and efficient ViT architectures. In a multi-stage stacked structure, the AGSAFormer reconstructs and optimizes basic modules, striking a balance between training efficiency, inference speed, computational cost, and model accuracy. The overall structure of the AGSAFormer follows the multistage and hierarchical representations. This approach facilitates multi-scale feature extraction while mitigating computational overhead. The basic AGSA block incorporates a window attention mechanism for local feature processing and a Gated Linear Unit for global feature processing. An adaptive feature processing block is employed to integrate global and local information, enhancing overall accuracy. Without using the teacher model and additional data, our efficient vision transformer serves as the backbone network, was trained on the widely adopted ImageNet-lK image classification dataset. Utilizing a single GPU, we completed 300 epochs of training in less than 48 hours. The model achieved a Top-l accuracy of 74.5%, surpassing existing baseline (EfficientVit). When deployed for inference in ONNX format on CPU, our model is <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$2\times$</tex> as fast as the baseline algorithm. In comparison to current larger-scale Transformer architectures and other state-of-theart CNN models, our efficient Vision Transformer exhibits competitive results, striking a balance between performance and efficiency.