Inferring Agent Speaking Styles for Auditory-Visual User-Agent Conversation

Conversational text-to-speech (TTS) aims to generate speech with proper style and emotion following the history of current dialogue catering to the demand of users. Meanwhile, human facial expression is a significant signal in conversation for intention and emotion estimation. In this paper, we introduce facial expressions in inferring speaking styles for agents in TTS applications, addressing the challenge of synthesizing speech with appropriate styles and emotions for conversation. We propose a multi-modal and multi-scale Transformer model utilizing visual and auditory features to infer styles in synthesized speech. Due to the insufficiency of publicly available visual-auditory conversational corpora, we establish a large-scale, high-quality multimodal dataset, Podcast, for this purpose. Experimental results demonstrate the proposed method outperforms the state-of-the-art in style inference and synthesizes speech coherent with conversation history.