Voice Cloning Method Based on Speaker Encoding with Global Timbre Constraints

Zero-shot voice cloning can rapidly synthesize speech resembling a target speaker using text input and a short reference audio sample within a few seconds. Current voice cloning techniques utilizing speaker encoding often fail to achieve adequate similarity to the target timbre when cloning the voices of unseen speakers. To address this problem, we propose a three-stage voice cloning architecture, where the core speaker encoder module leverages an enhanced TDNN-based x-vector model. It incorporates global timbre constraints, efficiently guiding the speaker features encoded by this model to maintain consistency both locally and globally. The proposed x-vector embedding with global timbre constraints (xv-gtc) is used as a condition in the end-to-end TTS model for voice cloning. To validate the embeddings’ capacity for the representation of speaker timbre, we propose an evaluation process for visualizing and analyzing embedding distributions. Additionally, we compare the effects of various embeddings as conditions from the perspectives of audible waveforms for voice cloning. Experiments show that the xv-gtc model has the optimal cosine similarity performance and embedding clustering effect in the speaker verification task, and the highest MOS values for naturalness and similarity. Compared to the baseline without the global timbre constraints, the naturalness MOS increased by $5.10 \%$, and the similarity MOS increased by $12.13 \%$.