Multi-Head AdaIN Audio-to-Video Motion Style Transfer

By leveraging the advancements in deep learning, video generation not only simplifies the creation of visual content but also enhances storytelling, thereby making dynamic visual experiences more accessible. The process of generating videos from audio input entails several key considerations, including frame quality, smooth motion and synchronisation with audio. While image quality is a fundamental aspect of image generation, creating smooth and synchronised motion trajectories is a challenging aspect of video generation that needs significant attention. Various approaches exist to learn video motion dynamics, such as representation decomposition and a self-supervised technique known as shuffling. However, these approaches have drawbacks such as motion artifacts due to video segmentation. To address this challenge, an alternative novel methodology is proposed that offers a comprehensive understanding of motion dynamics without dividing the generation process into multiple streams. This work leverages Adaptive Instance Normalisation (AdaIN) to adjust motion, enabling the generation of synchronised video sequences based on audio signals. The effectiveness of the proposed method has been validated through both qualitative and quantitative metrics, comparing with multiple models.