Dynamic Attention Fusion Decoder for Speech Recognition

Speech recognition serves as the foundation for human-computer interaction. To attain more accurate speech recognition results, the models for speech recognition are becoming increasingly sophisticated, demanding a greater volume of training data. In situations where data and computational resources are limited, the rapid development of usable speech recognition models becomes crucial for the future of this field. Currently, mainstream speech recognition models rely on attention mechanisms. These models employ cross-attention during decoding to address the relationship between the encoded speech input and the textual representation of the output. This mechanism allows the model to focus on different parts of one sequence while generating elements of another sequence, facilitating a better understanding of their relationships. However, stacking numerous layers of cross-attention is required to increase the model's expressive power. Yet, too many layers can impact model training and inference speed, and necessitate more data for training. We introduce a dynamic attention fusion speech recognition decoder designed for small datasets to address this issue. The decoder utilizes enhanced positional information during the decoding process to query the positional correspondence between the output text and the input speech encoding. This approach eliminates the need for extensive stacking of cross-attention mechanisms. Subsequently, a dynamic fusion module integrates these correspondences with the original decoding information. This method effectively establishes improved correspondences between the input and output, eliminating the need for stacking additional decoder layers and thereby reducing the model's parameter count. Our model achieved Character Error Rates (CER) of 4.60%, 12.67%, and 7.06% on the Aishell1, Primewords, and Free ST Chinese Mandarin Corpus datasets, respectively. Meanwhile, on the Uyghur dataset, our model attained a Word Error Rate (WER) of 4.23%. These results outperformed the baseline systems. The error rates decreased by 0.05%, 0.23%, 0.21%, and 1.52% on four datasets respectively compared to the baseline system. Additionally, our model's parameter count also decreased by 6.7%.