CoT-VTM: Visual-to-Music Generation with Chain-of-Thought Reasoning

The application of visual-to-music generation (VTM) is rapidly growing.However, current VTM methods struggle with capturing the relationship between visuals and music in open-domain settings, mainly due to two challenges: the lack of large-scale, high-quality visual-music paired datasets and the absence of direct semantic correspondence between visuals and music.In this work, we propose CoT-VTM, a framework that distills Chain-of-Thought (CoT) reasoning to enable visual-tomusic generation without paired data, while efficiently producing music aligned with visual content in open-domain settings.We first bridge the gap between visual, music, and text data using appropriate foundation models.Next, we identify key elements of the visual-music relationship and design a CoT prompt for visual-to-music mapping.To fully distill the reasoning of CoT, we incorporate latent information from intermediate reasoning steps as supervisory signals alongside visual and music supervision.Finally, we design a two-stage mapping distillation training process: the first stage uses discriminative MLP modules, while the second uses a generative embedding diffusion model (EmbedDiff).Our model achieves optimal performance on both imageto-music and video-to-music tasks.