Context-Aware Dual-Path Cross-Attention for Content-OrientedText-Audio Speech Source Separation

This paper proposes a transformer-based speech source separation model that leverages both audio and textual data to improve separation performance by incorporating contextual information from text. Speech separation enables the extraction of speech signals from audio mixtures containing background sources, facilitating downstream content repurposing. Textual information embedded in media content, such as scripts, provides contextual cues that can improve separation performance. To integrate such contextual cues, the proposed model employs a text embedding network to extract contextual and token-level representations, and integrates them into a transformer-based separation framework via cross-attention, enabling alignment between text and audio features. These features are then aggregated with the audio feature vector via cross-attention within a transformer-based separation model, which enables alignment between the two modalities. The proposed model is evaluated on simulated mixtures created from the LibriSpeech and MUSDB18 datasets. Experimental results demonstrate that the proposed model achieves performance improvements over existing text-audio separation models, as measured by SDR, SAR, SIR, PESQ, and STOI metrics. Furthermore, the proposed separation model achieves superior performance in scenarios involving vocal source-included background music, where conventional approaches typically degrade.