Video-driven cross-modal controlled music generation

Music plays a crucial role in video production, not only improving the overall quality of the video, but also enhancing the audience’s immersion. With the rapid growth of video content demand, the demand for cross modal music generation is also increasing. However, professional composers face high costs and complex production cycles, making it difficult to meet this market demand. Furthermore, current research has not effectively established and aligned the relationship between video and music as two modalities. To address the aforementioned issues, this paper proposes a Video-driven cross-modal controlled music generation (CPVC). This model aims to improve the quality of generated music by establishing a multi-level mapping relationship between video and music elements. Through the analysis of basic objective evaluation and comprehensive evaluation of controllability, the experiments show that the model in this paper generates samples that are closer to real music performances.