LLM-Based Text-To-Speech Generation Optimized Via a Dual-Branch Architecture

Large Language Models (LLMs) have significantly advanced text-to-speech (TTS) systems by leveraging discrete tokenization techniques. However, existing models struggle with three core limitations: (1) essential acoustic details are often lost due to the quantization of speech inputs; (2) the reliance on accurately aligned speech-text training pairs restricts scalability and practical usage; and (3) the process of adapting LLMs for speech token output tends to impair their original textual understanding, leading to catastrophic forgetting. To overcome these issues, we introduce a novel LLM-driven TTS framework named GOAT-TTS (Generation Optimized via dual-branch Architecture). The proposed architecture includes two main components: (1) a modality-alignment branch that integrates a speech encoder with a projection module to extract continuous acoustic features, enabling a robust link between semantic text and paralinguistic attributes like emotion, language, and voice quality without requiring textual transcripts; and (2) a speech-generation branch that performs targeted fine-tuning of the upper layers of an LLM for speech token output, while the lower layers are kept fixed to maintain linguistic understanding. Additionally, we incorporate a multi-token prediction mechanism to enable real-time, streaming TTS generation. Our empirical evaluations show that GOAT-TTS delivers results on par with leading TTS systems, with added benefits in generating diverse dialectal speech