An In-Depth Review of 2D-to-3D Deep Generative Models: from Workflow to Data Analysis

In recent years, there has been a notable increase in the development and scholarly publication of deep 3D generative models in computer vision, and among them, the 2D-to-3D deep generative models stand out to be one of the most popular types, such as AG3D, CC3D, GINA3D, RODIN, GNVS and so on. To give a brief definition, 2D-to-3D deep generative models can process 2D input through a series of data transformations inside the model based on deep learning methods and eventually produce the desired 3D representation, such as 3D human avatars, 3D human models, 3D scenes, and 3D objects. 3D deep generative models are in huge need in the global market because of their wide variety of productions that can be incorporated into relevant industries. For instance, 2D-to-3D deep generative models can make use of a combination of Generative Adversarial Network (GAN), Multi-Layer Perceptron (MLP), and Volume Rendering to generate 3D scene; Deep Convolutional Neural Network (DCNN) + MLP + Volume Rendering + Autoregressive Generation or Vision Transformer (ViT) + Cross Attention Module (CAM) + Volume Rendering to produce 3D object; Canonical Generator + MLP + Volume Rendering to adversarially generate 3D human model; Diffusion Model + MLP + Volume Rendering to produce 3D human avatar. This paper aims to present a categorization of the State-of-the-Art 2D-to-3D deep generative models and provide an in-depth review of the techniques utilized by these models based on workflow, code-reviewing, and data analysis.