Time matters: enhancing speech recognition with self-supervised audio reordering tasks

This paper proposes a novel self-supervised learning framework integrated with an end-to-end speech recognition model to enhance the encoder’s ability to capture the intrinsic features and structural dependencies of speech signals. The core idea is to design a self-supervised task that randomly shuffles segments of the input audio signal, training the model to restore them to their correct temporal order. This task strengthens the encoder’s capacity to model temporal dependencies and local features of speech signals. By jointly optimizing the self-supervised task and the speech recognition task within a shared encoder, the model effectively leverages labeled data for supervised learning while improving the robustness and generalization of feature representation through self- supervised learning. Experimental results on the AISHELL-1 and THCHS-30 datasets validate the effectiveness and generalizability of the proposed approach.