MVLLaVA: An Intelligent Agent for Unified and Flexible Novel View Synthesis

Novel view synthesis creates images from unseen angles, which is vital in computer vision. Recently, multi-view diffusion models have emerged as a powerful tool to synthesize views efficiently from a single prompt such as an image or a caption. However, these models lack the versatility and intuitive user control, which cannot achieve free-modal inputs and are restricted to their predefined modal. To overcome these limitations, we introduce MVLLaVA, an intelligent agent capable of handling five input types for novel view synthesis tasks. By integrating multiple diffusion models with LLaVA, MVLLaVA offers exceptional flexibility, adapting to a wide range of inputs and instructions. The careful design and efficient fine-tuning of task-specific instruction templates further enhance its performance across various scenarios. Experimental results validate the robustness and versatility of MVLLaVA, highlighting its potential as a powerful solution for unified multi-view generation tasks.