Hola-TTS: A Cross-Lingual Zero-Shot Text-to-Speech System for Chinese, English, Japanese, and Korean

This paper presents Hola-TTS, a novel cross-lingual zero-shot text-to-speech system capable of synthesizing speech in Chinese, English, Japanese, and Korean. We introduce several key innovations, including a comprehensive data cleaning process, a multilingual phoneme representation strategy, and an improved model architecture. Hola-TTS employs a fully autoregressive method for token generation and utilizes a delay parttern approach, which significantly enhance dependencies between different codebooks at the same time step. Our model is trained on a diverse set of cleaned speech recognition datasets and uses phoneme-level language label to enhance cross-lingual synthesis capabilities. Comparative evaluations demonstrate that Hola-TTS outperforms state-of-the-art models like VALL-E X and X-TTS across various metrics. Ablation studies further confirm the effectiveness of our proposed improvements compare to VoiceCraft. This work represents a significant advancement in multilingual speech synthesis, offering improved quality, consistency, and cross-lingual capabilities in zero-shot TTS. We encourage readers to listen to the demos at https://holatts.aithub.io/