Convincing Audio Generation Based on LLM and Speech Tokenization

This report describes the work submitted by the ZYZX-AI team to ICAGC 2024. This challenge, titled “Inspirational and Convincing Audio Generation Challenge 2024,” is part of the ISC-SLP 2024 Competitions and Challenges track, aims to enhance the persuasiveness and acceptability of synthesized audio, focusing on human alignment convincing and inspirational audio generation. Our solution combines speech discretization, a language modeling approach, and waveform reconstruction by a vocoder based on speech tokens. First, the input audio is mapped to a sequence of discrete tokens, audio generation as a language modeling task in this representational space. Sub-sequently, a neural audio codec uses the predicted discrete speech tokens in conjunction with the text to be synthesized to achieve high-quality audio generation. These models are trained on diverse datasets including open-source corpora such as wenetspeech4TTS and magicData. Unlike speech recognition tasks, speech synthesis tasks require higher-quality annotations in their training datasets. To address this issue, methods such as speech enhancement, loudness processing, and adjustments to punctuation in text annotations are applied during the data preprocessing stage to improve data quality. Our submission achieved third place on the track1 (Inspirational Emotion Modulation), with an average MOS value of 3.56.