A comprehensive analysis of the impact of Transformer model components and their interactions on overall model performance

The Transformer model is based on attentional mechanisms to process sequence information independently and utilises positional encoding to add sequence order. In addition to these two features, the model includes various components, such as its layered architecture and a greater number of hyperparameters compared to traditional machine learning methods. These components enhance effective data processing, but they require substantial time and computational resources for training. Therefore, traditional Transformer studies have focused on a limited subset of components. This study addresses this limitation by assessing the impact of a broader set of Transformer components, both alone and through their interactions, on the model’s overall performance. To overcome the challenge of testing all possible combinations of components, a fractional factorial design was used. This design reduced 256 possible combinations to 16 experimental runs, each performed twice for statistical reliability, while assessing batch sizes of 32 and 64. Through these analyses, the overarching conclusions derived from this study identify several key factors influencing performance: optimiser selection, batch size, and critical interactions between embedding dimension, positional encoding, and the number of attention heads. This study provides a significant advance in assessing the effects of the Transformer components, providing a more comprehensive approach compared to traditional methods. The aim is to support the explainable analysis of Transformer configurations.