Exploring Width-Adaptive Transformers for Automatic Speech Recognition

Transformer architectures with multiple heads with wide attention dimensions (widths) are over-parameterized. This leads to parameter redundancy and high correlations across attention heads with only a minority of heads actively contributing to the task. In this study, we quantitatively analyze the parameter redundancy by comparing the linear centered kernel alignment (CKA) similarity of learned representations extracted across attention layers and heads. Observing that widening the network can exacerbate these correlations, leading to representations with high CKA similarity, we question the design choice with uniform attention widths across all attention heads or layers and investigate how this choice impacts correlations across heads in the same layer. We design a width-adaptive training method to dynamically tune the model to keep the main contributing widths in each attention head and layer while no knowledge distillation or re-training process is needed. Experimental results on both English and Dutch corpora show our adaptive training method effectively reduces cross-head correlations and improves accuracy in automatic speech recognition. We also demonstrate the effectiveness of width-adaptive training by finetuning the OWSM speech foundation model.