Multi-Head Attention-Enhanced Speech Recognition for Reduced Data Requirements

Automatic speech recognition (ASR) technology has reached a mature level, and improving performance in data-scarce scenarios has become a key research focus. In this study, we propose a novel approach to building acoustic models by leveraging the multi-head attention mechanism to enhance various acoustic features extracted from audio data, combined with time-delay neural networks (TDNNs) using semi-orthogonal low-rank matrix factorization, resulting in significant performance improvements. To increase the robustness of the acoustic model, we introduce a small amount of data perturbation during the early stages of training. Additionally, during the decoding phase, we employ an external recurrent neural network language model (RNNLM) for rescoring, further enhancing the model’s accuracy. A comprehensive evaluation of the widely used LibriSpeech corpus shows that our method, using only half the training data, achieves word error rates (WERs) of 3.15 and 7.04 on the test_clean and test_other datasets, respectively. These results outperform models with similar architectures trained on the full dataset and demonstrate performance comparable to mainstream end-to-end models.