Controllable Latent Diffusion-Based 3D Brain Tumor Segmentation: With Synthetic Label Generation and Detailed Variance Map

Approaches based on the Denoising Diffusion Probabilistic Model (DDPM) have shown promise for directly generating segmentation maps from medical images. However, denoising in the original image space limits the application of DDPM to 2D images. We present a latent diffusion model-based segmentation method (LDM-seg) to directly generate multi-label segmentation maps from 3D medical images, such as multisequence magnetic resonance imaging (MRI). A distinctive aspect of our approach is utilizing ControlN et to apply MRI as a conditioning factor to control the generation process. Trained and validated on the BraTS 2023 Adult Glioma dataset, we show LDM-seg outperforms state-of-the-art methods, including nnU-Net and MedNeXt. In addition to segmentation, the method can be used to generate an unlimited number of realistic brain tumor masks, which are typically required as conditions for generating synthetic brain MRI with tumors. Further, the method can also produce a detailed variance map of predicted segmentations.