Storytelling Style Speech Generation System: Emotional Voice Conversion Module Based on Cycle-Consistent Generative Adversarial Networks

Telling a story requires various emotional ups and downs as well as pauses. Preparing a parallel corpus for emotional voice conversion is often costly and impractical. Developing high-quality non-parallel methods poses a significant challenge. Although non-parallel methods have been shown to enable emotional voice conversion, its capability for Chinese storytelling has not been clarified. Additionally, the storytelling results of emotional voice conversion have not been validated within a 3-12-year-old children. This study proposes a two-stage Chinese Storytelling Style Speech Generation System (SSPGS) composed of a text-to-speech system and an emotional voice conversion module. The SSPGS requires no parallel utterances, transcriptions, or time alignment procedures for speech generator training and requires only several minutes of training examples to generate reasonably realistic sounding speech. A small corpus neutral speech model is constructed on the text-to-speech system in the first stage, which is based on the speech synthesis system using a Hidden Markov Model (HMM). In the second stage, the emotional voice conversion module based on Cycle-Consistent generative adversarial networks (CycleGAN) is built. It enables the neutral speech generated by the text-to-speech system in the first stage to be transformed into the happiness, anger, and sadness necessary for storytelling tone using the timbre (spectrum), pitch (fundamental frequency F0), and rhythm (speech rate) of neutral speech. The validity of SSPGS is verified in two ways. First, a 5-point Mean Opinion Score (MOS) was performed for young children’s parents. The results demonstrated that compared with general speech synthesizers, such as Google, the system generated more natural and genuine sound, that was more preferrable to the target audience. After that, the kids underwent a story immersion evaluation. Analysis of the degree of engagement, liking, and empathy in listening to the story revealed no statistically significant difference between real-person dubbing and emotional speech synthesis dubbing. As a result, it has been initially confirmed that SSPGS might be added to the story robot product in the future.