Inferring Speaking Styles for Conversational Speech Synthesis by Learning Contextual Dependencies

In addition to conventional text-to-speech (TTS) systems, conversational TTS systems need to infer the appropriate speaking style on the basis of the current conversational scenario and the dependencies within the context. These dependencies specifically include the interrelationships among speakers as well as the multimodal information present in the context at both the global scale (i.e., the utterance level) and the local scale (i.e., the word level). However, current context modeling techniques in conversational TTS focus only on global-scale modeling of textual information, disregarding the modeling of speaker dependencies and other fine-grained and multimodal features in context. In this paper, we propose several context modeling techniques to enhance context modeling and the inference of speaking styles in conversational TTS. We propose approaches inspired by DialogueRNN and DialogueGCN to capture inter- and intraspeaker dependencies through textual and acoustic features in conversations. We further propose an approach based on a multiscale relational graph convolutional network (MSRGCN) to learn the fine-grained dependencies at the local scale. The experimental results demonstrate the effectiveness of our proposed context modeling techniques compared with traditional methods in conversational TTS systems. The contributions of modeling multimodal information and multiscale dependencies are further demonstrated in ablation studies.