Addressing human speech characteristics in single-channel speaker extraction networks

The objective of single-channel speaker extraction is to isolate the clean speech of a target speaker from a mixture of multiple speakers’ utterances. Conventionally, an auxiliary reference network is utilized to extract the speaker’s voiceprint features from the speech signal, which are then fed as input cues to the primary speech extraction network, thereby enhancing the extraction robustness. Nevertheless, existing studies have largely overlooked the sequential characteristics of speech signals, leading to a mismatch between the receptive field of the model and the inherent signal features. Moreover, prevalent speaker extraction architectures fail to account for the spectral distribution properties of speaker speech. To address these limitations, this study presents an extended version of our previous work ( DOI 10.36227/techrxiv.23849361.2 ), which proposes an innovative approach by extending the Convolutional Neural Network Next (ConvNeXt) model, originally designed for image processing, into the Time-Domain ConvNeXt model (TD-ConvNeXt). The TD-ConvNeXt model is integrated with Temporal Convolutional Networks (TCN) blocks to construct the core architecture of the speech extraction network. Additionally, the ConvNeXt model is adapted into a novel Spk block for the auxiliary network, which effectively learns the identity-related features from the reference speech and represents them as embedding vectors. This methodology enables high-fidelity extraction of the target speaker’s speech while maintaining excellent speech quality. The extraction network and the auxiliary reference network are jointly optimized via multi-task learning. Extensive experimental evaluations demonstrate that the proposed model substantially outperforms state-of-the-art methods in single-channel target speech extraction. Across various signal-to-noise ratios (SNRs), the proposed model achieves superior performance in terms of Scale-Invariant Signal-to-Distortion Ratio (SI-SDR), Perceptual Evaluation of Speech Quality (PESQ), and Short-Time Objective Intelligibility (STOI). Specifically, at an SNR of 5 dB, the proposed model attains SI-SDR, PESQ, and STOI scores of 14.83, 3.03, and 0.934, respectively.