Enhancing Multi-Speaker Audio Generation with Limited Samples Based on Acoustic Signals

His paper presents a method for enhancing multi-speaker audio generation using limited samples based on acoustic signals. The proposed approach improves the existing variational inference text-to-speech (VITS) model by incorporating the vector quantized variational autoencoder (VQVAE) network and the attention mechanism to enhance audio generation quality for different speakers. Traditional VITS models trained on large and high-quality datasets can generate high-quality text-to-speech outputs with good generalization. However, with small datasets, the VITS model often struggles to maintain audio quality and speaker characteristics. To address these limitations, we introduce the VQVAE network to discretize continuous acoustic features into tokens, effectively capturing essential speaker characteristics such as timbre and articulation. The attention mechanism further improves the model's ability to focus on these critical speaker-specific features, especially in multi-speaker scenarios. Experimental results show that our enhanced model not only excels in generating natural and high-quality speech for the original speaker but also effectively generalizes to other speakers. This method demonstrates significant improvements in naturalness and speaker characteristic retention, making it particularly effective for multi-speaker audio generation in scenarios with limited data.