Dance Motion-Guided Music Generation via Residual Vector Quantization

Music is a traditional and important element in human entertainment. We propose a novel deep learning-based method for generating music from human dance movements. We employ Residual Vector Quantization (RVQ) for music tokenization, using RVQ indices as the music representation, thereby reducing learning complexity. A cross-modal generation model integrating LSTM and attention mechanisms is designed to generate RVQ codebook indices from dance motions. Finally, the indices can be converted back to music waveforms through a music decoder. Experimental results demonstrate the feasibility of generating coherent music that aligns with dance dynamics, providing a new approach for cross-modal entertainment content creation.