Improved Compact Convolutional Transformers: The Faster Convergence Resource-Efficient Transformers

Despite the excellent performance of Transformer-based architectures, they also suffer from the drawback of high computational resource consumption. To address this issue, Compact Convolutional Transformers(CCT) have achieved strong competitiveness on small image datasets with fewer parameters through appropriate sizing adjustments and convolutional tokenization. Based on this, this paper further proposes Improved Compact Convolutional Transformers(ICCT). These models have achieved a comprehensive improvement in accuracy compared to CCT by modifying their positional encoding and feed-forward neural network, while keeping all other components, parmeters, and hyperparameters unchanged.