GenMedia: A Research Project on Personalized Multi-Modal Media Generation Using Stable Diffusion and AudioCraft

In this paper, we present GenMedia, a research project focused on advancing the capabilities of Stable Diffusion models and AudioCraft's MusicGen for personalized multi-modal media generation. The project explores the fine-tuning of Stable Diffusion 2.0 and 3.5 using DreamBooth, with a particular emphasis on advancements in latent space optimization, prompt adherence, and image quality. These improvements enable the generation of high-quality, context-aware images based on user-provided text prompts and personalized datasets. Additionally, we investigate the fine-tuning of AudioCraft's MusicGen using Dora to synthesize personalized audio content, leveraging custom instrumental datasets and text prompts. The integration of FFMPEG enables the seamless combination of generated images and audio into cohesive video outputs. Through extensive experiments, we evaluate the performance of these models, focusing on their ability to create highly personalized and contextually relevant media content. This research highlights the potential of advanced generative models to revolutionize multi-modal AI, paving the way for future innovations in personalized media generation.