Natural-Prosodic Cross-Lingual Personalized TTS

Despite long-standing efforts in speech synthesis technology, several shortcomings persist, including unnatural pronunciation, rigid cross-lingual synthesis, the presence of synthesized noise artifacts, and inadequate synthesis speed. Customizing a speech synthesis system to accurately mimic the timbre, intonation, rhythm, and prosody of specific speakers remains a significant challenge. To address these deficiencies, we present an advanced text-to-speech (TTS) AI model that seamlessly integrates autoregressive modeling, BERT, HuBERT, and self-supervised learning within the VITS framework. This model, referred to as ABS-VITS, enhances VITS by enabling high-fidelity synthesis that more realistically mimics human speech. The BERT component effectively extracts features necessary for reconstructing vocal timbre, while randomized duration prediction generates speech with diverse linguistic rhythms—critical elements for simulating human pronunciation. By strategically organizing and applying these pretrained models, ABS-VITS achieves a high degree of accuracy in producing speech waveforms from text inputs. Notably, the proposed network framework allows users to select their desired target speaker, significantly enhancing the naturalness of the synthesized speech after rapid fine-tuning. Consequently, this model facilitates personalized cross-lingual speech synthesis that emulates vivid prosodic emotions.