Taming Stable Diffusion for MRI Cross-Modality Translation

In this study, we explore using Stable Diffusion (SD) for unsupervised medical image-to-image translation. SD has shown remarkable performances in generating high-quality images and can be easily applied to generate custom contents by injecting standard plug-ins like LoRA, offering a promising solution to tackle the complexity caused by variations in imaging modalities, acquisition parameters, and body parts in medical imaging. However, We empirically find that existing pipelines designed for natural images fail to translate directly to medical images due to weak structural control and inappropriate color preservation. To address these issues, we propose a novel two-branch image translation pipeline. This pipeline decouples the generation of target image along the time axis and employs ControlNet to ensure precise structural preservation. Additionally, we customize SD to generate images of extreme brightness, a common feature in medical imaging. Our results on the BraTS dataset demonstrate that SD with task-specific plug-ins can generate high-quality medical images comparable to those generated by task-specific models. Since the development of these standard plug-ins can be easily done by clinicians without much knowledge of the underlying algorithm, such a mode holds the potential to significantly extend the use of medical image computing algorithms in the clinical environment.