Fusion of Features from Multiple ASR Pretrained Models: Enhancing Robustness and Accuracy

Compared to traditional feature extraction methods like MFCC and FBank, self-supervised learning (SSL) based pre-trained models in automatic speech recognition (ASR) can capture deeper speech patterns and contextual information, thereby improving recognition accuracy and robustness. Given the significant success of individual SSL models, we explored whether fusing features from multiple pre-trained models, such as Wav2Vec2 and HuBERT, could further enhance ASR performance. Using the AISHELL-1 dataset, we performed weighted summation of the outputs from different layers of each SSL model and employed a multi-head cross-attention mechanism to fuse features from different models, enabling the model to capture complementary information from multiple sources. The fused features were then input into a Conformer Encoder and CTC/Attention Decoder network. The experimental results show that multi-model feature fusion significantly improves system performance.