Retrieve-and-Generate: Image-to-3D Framework with Mesh Retrieval and Texture Generation

In recent years, significant advancements have been made in the field of image-to-3D reconstruction. However, one of the persistent challenges that remains unresolved is messy surfaces of generated outputs. To tackle this issue, we introduce a novel retrieve-and-generate scheme specifically designed for image-to-3D reconstruction tasks. Our method involves a two-stage process: first, the model retrieves the most similar 3D mesh corresponding to the input image; second, it generates a texture map that can be accurately mapped onto the retrieved mesh. In the initial retrieval phase, our approach leverages a pre-trained multi-modal joint representation to identify the 3D mesh that closely resembles the input image within the embedding space. Subsequently, texture generation module generates a realistic texture reflecting the input image, which leads to the complete 3D object reconstruction when this texture is mapped to the retrieved mesh. We have observed that our retrieve-and-generate approach significantly enhances the quality of the reconstructed 3D objects from a single input image. This improvement in reconstruction performance demonstrates the efficacy of our proposed method and its potential to advance the state-of-the-art in image-to-3D reconstruction technology.