HoloDiT: Exploring 3D Diffusion Transformer Models Using 2D Images

Recently, Transformer-based diffusion models have produced impressive results in 3D shape generation. However, at this stage, they can only be trained using 3D point cloud data to generate simple 3D shapes based on point cloud representations. Building on this, we propose HoloDiT. It uses a set of real-world 2D images and corresponding camera poses as input, generating 3D auxiliary samples that are fed into a Transformer-based 3D diffusion model for training in 3D shape generation. By training the rendering function to generate color and density, we can restore the color details of object surfaces. This enables HoloDiT to generate realistic 3D shapes from 2D images. We trained on the real-world CO3Dv2 dataset, and the generated results demonstrate competitive quality and consistency compared to existing 3D generation models.