Emotional Speech Synthesis Based on Gated-Style Tokens and A Rhythm Predictor

Currently, speech synthesis has found increasingly widespread applications across various fields. Although the quality of synthesized speech continues to improve, challenges remain in emotional speech synthesis, particularly in achieving naturalness and authentic emotional expression. Additionally, traditional emotional speech synthesis models often suffer from low generation efficiency due to their complex processing pipelines. To address these two issues, a LightGrad-based emotional speech synthesis model is proposed in this paper, featuring a constructed feature fusion module, along with a gated style token layer and a prosody predictor incorporated into the model. The gated style token layer helps generate emotional style embedding vectors to capture more subtle and complex emotional details. The prosody predictor provides prosodic information for audio generation to improve the rhythm, pausing patterns, and natural transitions of intonation in speech. In addition, dilated causal convolutions are used to replace depthwise separable convolutions, enhancing the model's ability to model sequential data, while optimal transport conditional flow matching (OT-CFM) is employed to improve the diffusion decoder, effectively increasing the synthesis efficiency of the model. Experimental results demonstrate that the proposed model can generate emotionally expressive speech while maintaining high synthesis quality. Moreover, training on both English and Chinese corpora verifies its cross-lingual capability.