UniTMGE: Uniform Text-Motion Generation and Editing Model via Diffusion

Current methods have shown promising results in applying diffusion models to motion generation given text input. However, these methods are limited to unimodal inputs and outputs, restricted to motion generation alone, and lacking multimodal control capabilities. To address these issues, we introduce UniTMGE, a text-motion multimodal generation and editing framework based on diffusion. UniTMGE overcomes single-modality limitations, enabling exceptional performance and strong generalization across multiple tasks like text-driven motion generation, motion captioning, motion completion, and multimodal motion editing. UniTMGE comprises three components: UTMV for mapping text and motion into a shared latent space using contrastive learning, a controllable diffusion model customized for the UTMV space, and MCRE for unifying multimodal conditions into CLIP representations, enabling precise multimodal control and flexible motion editing through simple linear operations. We conducted both closed-world experiments and open-world experiments using the Motion-X dataset with detailed text descriptions, with results demonstrating our model's effectiveness and generalizability across multiple tasks.