Self-supervised Generative Network for Personalized Speech Enhancement

Lack of accurate array information and massive labeled training sets of speakers makes it impossible to learn in a supervised manner for speech enhancement in realistic unseen scenes. To address this problem, we proposed a self-supervised generative adversarial framework based on speaker verification posterior probability. Considering the sparsity of sounds, three-dimensional mask is applied with the generator to reconstruct signals from multiple sources, without requiring the clean speech of the target. A speaker verification model is applied as the discriminator to identify features related to the target speaker based on only few-shot samples. Adversarial training is performed with speaker verification posterior probability, reconstruction loss and channel-independent loss for personalized speech enhancement. Experiments demonstrate that proposed framework can outperform baselines, and our self-supervised speech enhancement can achieve comparable performance with supervised methods.