Doodle Your Motion: Sketch-Guided Human Motion Generation

Recently, significant progress has been made in condition-guided human motion generation. However, due to the inherent abstraction of conditional semantics like text, music and trajectory, these methods often fall short of generating precise motions that align with human intent. In contrast, free-hand sketches inherently and accurately depict human perspective intent, finding extensive applications across multiple domains. In this article, we introduce Sketch-guided human Motion Diffusion (SMD), to address a novel scenario: sketch-to-motion, aiming to generate plausible and natural human motions based on human motion sketches. Specifically, Our proposed SMD employs a Dual-branch Time-aware Transformer that utilizes both global semantic and local perspective level attention to condition 2D sketch information for 3D motion generation. At the global semantic level, we establish associations between the representation of the entire sketch and the sequential motion to ensure the generated motion aligns with the semantic content of the sketch. Meanwhile, at the local perspective level, a sketch-aware local attention is devised to correlate the sketch patches with the motion keyframes, aiming to precisely align the keyframes with the given sketches. Rooted in Diffusion model and Dual-branch Time-aware Transformer, our approach demonstrates proficiency in motion in-betweening and body part editing tasks, seamlessly generating natural motion sequences that harmonize with the provided context. Multiple experiments conducted on the curated sketch-to-motion datasets validate the efficacy of SMD, showcasing the state-of-the-art generation performances.