What to Refer and How? - Exploring Handling of Auxiliary Information in Target Speaker Extraction

This study explored the appropriate auxiliary information for target speaker extraction (TSE). In TSE, the conventional method involves using pre-recorded speech as an auxiliary signal. However, in multi-channel environments where spatial information is available, alternative approaches that utilize a rough pre-extraction of the target speech as an auxiliary signal are being explored. Pre-recorded speech is typically clean but contains different content, whereas pre-extracted speech has content aligned with the target but includes various types of distortion depending on the extraction method used. These two approaches differ in the characteristics of the auxiliary signals and the conditioning methods applied. Previous research has not thoroughly examined the relationship between the nature of auxiliary signals and TSE model performance. This study aims to explore the effective use of auxiliary signals in TSE models and to provide a comprehensive analysis of their impact. Through TSE experiments conducted in a multi-channel environment, we investigated the effects of auxiliary signals from three key perspectives: the trade-off between content consistency and distortion, the level of referencing, and the type of distortion.