Image-Conditioned 3D Content Creation Based on 2D Diffusion

Creating the 3D geometric model from the single monocular image is a severely under-constrained problem, due to that the single image alone is insufficient to determine both the scale and position of an object. To tackle this problem, we present a two-stage formulation to generate 3D geometry from a single RGB image. First, we design the view-conditioned latent diffusion model which takes in a monocular image along with camera extrinsic to generate multi-view images. Then, we train the neural radiance field based on the generated multi-view images and adopt the marching cube algorithm on the learned density field to produce the geometric model. This approach eliminates the need for massive explicit 3D supervision. Experimental results showcase the effectiveness and strength of our framework.