Low-Latency Single-Microphone Speaker Separation with Temporal Convolutional Networks Using Speaker Representations

This article introduces a novel method for single-channel speaker separation, focusing on causal and low-latency inference. While significant advancements have been recently made in speaker separation, especially in non-causal scenarios, there is a notable gap concerning low-latency speaker separation — a critical requirement for real-time conversational applications like phone calls, video calls, and human-machine interaction. We propose a two-stage solution that leverages initial-separation in conjunction with speaker representation-driven refinement. To assess the effectiveness of our method, we conducted extensive experiments and examined its performance in both a very low algorithmic latency of 16 milliseconds and a fully causal model. Additionally, we delve into a detailed analysis of the composition of datasets derived from WSJ0, shedding light on their impact on model evaluation. Our two-stage solution demonstrates significant performance enhancements compared to the baseline model and offers an easily deployable solution suitable for edge devices.