A Unified Approach to Speaker Separation and Target Speaker Extraction Using Encoder-Decoder Based Attractors

Blind speaker separation (SS) and target speaker extraction (TSE) deal with the extraction of speaker signals from mixtures containing multiple speakers. We propose a unified approach to SS and TSE that requires no explicit training for the TSE task. The considered SS system is based on SepFormer and trained for joint speaker counting and source separation using an internal separation scheme based on Encoder-Decoder-Attractors (EDAs). We train the system in a multi-task learning setup with an additional head for speaker identification using attractors from the EDA module. We show that it is then possible to perform inference in the TSE mode without explicitly extracting all the sources. Furthermore, we show that it is possible to detect the presence or absence of the target speaker. Evaluation using the synthetic speech mixtures shows the feasibility of the proposed unified SS and TSE system with matching performance in both configurations.