Towards Releasing ViT from Pre-training

Vision Transformers (ViT) have revolutionized computer vision by leveraging self-attention mechanisms to process images. However, their pre-training requirements can be expensive or impractical for many applications. In response, researchers have proposed various architectures and modifications to improve efficiency and accuracy. This study systematically investigates and evaluates the performance of different Vision Transformer architectures in order to enrich the analysis, including pure and hybrid models, on three benchmark datasets. We propose a comprehensive evaluation that considers various architectural modifications, such as integrating convolutional layers and attaching different classification heads. Our experiments show that incorporating convolutional layers into ViT models and replacing the classification head with an average token approach can lead to substantial improvements in performance when trained on small datasets without pre-training.