Scaling Up the Transformers: A Survey of Training and Inference Optimization Techniques

Transformer models have shown great success in Natural Language Processing (NLP) tasks, but they come with unique challenges such as the vanishing gradient, unbalanced gradient, and computational costs. To overcome these challenges, various optimization techniques have been developed including regularization, parallelism, and hybrid architectures. In this survey paper, we review the recent advances in optimizing transformers for NLP tasks and highlight their applications on weakly labelled data and low-resource settings. We discuss how to improve the performance of Transformers on Convolutional Neural Network (CNN)-Transformer models, which leverage both CNN and Transformer architectures. Furthermore, we explore techniques specifically designed for weak supervision and low-resource scenarios such as Segment-wise training, Cluster-based attention mechanism, and Transfer learning from a teacher model to a student model. We also highlight some recent works on Sound Event Detection using CNN-Transformer, which leverages the strength of CNNs for feature extraction and Transformers for long range dependencies. Overall, this paper provides an overview of the challenges faced during optimization of transformers in NLP tasks and offers insights into the state-of-the-art techniques to overcome these challenges.