From Pixels to Volumes: Generative AI in 3D Medical Imaging

With the advent of generative AI, medical imaging has been revolutionized, allowing for unprecedented capabilities in data generation, volumetric reconstruction, and clinical decision support. Although significant advances have been achieved in two-dimensional modalities, extending them to three-dimensional medical imaging, such as Magnetic Resonance Imaging (MRI), Computed Tomography (CT), and Positron Emission Tomography (PET), remains a developing frontier with new technical and clinical challenges. This survey offers a thorough, systematic exploration of generative AI approaches uniquely applicable to 3D medical imaging, including voxel-based generative models, implicit neural representations, and latent diffusion models. The literature is organized in three orthogonal axes: imaging modality (MRI, CT, and PET), model architecture (GAN, VAE, diffusion, and NeRF), and clinical application (augmentation, reconstruction, surgical planning, and anomaly detection). We provide detailed taxonomy tables for each axis, including landmark papers, strengths, limitations, key techniques, and benchmark performance. We also address evaluation protocols, ethical issues related to synthetic data, and open research challenges. We analyzed more than 50 representative works and found that latent diffusion models have firmly established themselves as the standard for high-fidelity 3D synthesis and that implicit neural 3D representations are best for reconstructing 3D scenes from sparse views with limited memory. It does not, however, mean that they perform better across all modalities and tasks, as they have significantly greater requirements in terms of computational and memory load, sampling time, and training data compared to alternatives like diffusion-based methods (which account for about 44% of the surveyed landmark architectures. Finally, we propose a clinical, ethical, and technically sound blueprint for the use of generative AI in volumetric medical imaging.