MDG:Multilingual Co-speech Gesture Generation with Low-level Audio Representation and Diffusion Models

Co-speech gesture generation is crucial for enhancing human-computer interaction. Most existing research focuses on English, overlooking multilingual speech and the ability to generalize across languages. Additionally, these studies often rely on pre-trained audio feature extraction models, insufficient for non-English languages. To address these issues, we introduce MDG, a novel diffusion model-based method for speech-driven gesture generation using a multilingual dataset. Our approach incorporates Language ID to distinguish between languages and utilizes low-level audio features to improve multilingual generalization. Extensive experiments demonstrate that MDG significantly outperforms baseline models in multilingual co-speech gesture generation. For more details, please visit our demo page<sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup>.