Generating multiview-consistent images from a single-view image based on latent diffusion model

In recent years, latent diffusion models (LDMs) have successfully demonstrated their strong generative and generalization capabilities in the field of image generation. However, when applied to generate multiple-view image of the same object, LDMs are unable to ensure the 3D-consistencies among the generated multiple images. Existing multiple-view generation methods leverages the power of CLIP image embedding combined with camera embedding to produce semantically coherent images. However, such semantic multi-view consistent heuristics is rather weak and does not necessarily follow any shared 3D structure, which leads to geometric and color inconsistencies in the generated images. To address this issue, in this paper we propose a consistency model (named ControlGAM) based on camera extrinsics. The core of our model is a 3D feature attention mechanism that exchanges 3D information during the diffusion of multiple single views according to basic multi-view geometric principles. Experimental results on the Google Scan dataset demonstrate that ControGAM can generates highly consistent images at different views, making it suitable for a variety of generative tasks such as new view synthesis, text-to-3D and image-to-3D.