Trainable multi-channel front-ends for joint beamforming and speaker embedding extraction

Multi-channel speaker verification (SV), employing numerous microphones for capturing enrollment and/or test recordings, gained attention for its benefits in far-field scenarios. While some studies approach the problem by designing multi-channel embedding extractors, we focus on building and thoroughly analyzing a framework integrating beamforming pre-processing paired with single-channel embedding extraction. This strategy benefits from accommodating both multi-channel and single-channel inputs. Furthermore, it provides human-interpretable intermediate output — enhanced speech — that can be independently evaluated and related to SV performance. We first focus on the front-end, taking advantage of deep-learning source separation for direct or indirect mask estimation required by the beamformer. We alternate single-channel network architectures, subsequently extended to multi-channel ones by reference channel attention (RCA). We also analyze the impact of beamformer and network output fusion. Finally, we show improvements brought by end-to-end fine-tuning the entire architecture facilitated by our newly designed multi-channel corpus, MultiSV2, extending our previous MultiSV dataset. • Multi-channel speaker verification with advanced front-ends improves performance. • Reference channel attention extends single-channel separators to multiple channels. • Joint training of the pre-processing and embedding extractors boosts performance. • Fusion of neural-network-based separator and beamformer outputs benefits separation. • MultiSV2 is a multi-channel simulated corpus with speaker labels and 1k h of speech.