Exploring the Impact of Activation Functions on Vision Transformer Performance

Vision Transformers are increasingly being utilized across various application domains and demonstrate performance in image classification tasks that is on par with state-of-the-art CNN models.An important aspect in optimization, fine-tuning, as well as the exploration of architectures, is the selection of activation functions.Activation functions introduce non-linearity to the models and are crucial components of all deep learning models.Regularly, new activation functions are published, promising to elevate the accuracy of the models to new levels.Commonly, researchers, impressed by successes on selected examples, eagerly adopt these new activation operators and integrate them into their models.However, what is the actual impact of these functions on training?In the context of an empirical study, we investigate the effect of using a wide range of activation functions on the accuracy and training time of selected vision transformers belonging to different categories.The results demonstrate, among other findings, that in addition to ReLU, activation functions from the Sigmoid family exhibit particularly favorable performance metrics.