Semi-Process Noise Distillation for Continual Mixture-of-Experts Diffusion Models

Generative diffusion models have made significant improvements in the image, audio, and video domains. However, existing diffusion models suffer from a critical challenge: after being trained on new datasets, they tend to forget how to generate images of the previous datasets domain. We name it as catastrophic forgetting of diffusion models. To address this challenge, we propose the NDE-Diff, a method that supports diffusion models trained in continual learning scenarios. The NDE-Diff contains semi-process noise distillation and memory augmented continual mixture of experts. Unlike conventional images replay distillation, our method distills the memory of the previous diffusion model with Gaussian noise during the intermediate denoising process. The memory augmented continual mixture of experts further increases the memory capacity of the diffusion model and introduces a task-guided routing network, effectively activating the utilization rate of the experts. Comprehensive experimental results demonstrate that our NDE-Diff significantly reduces training time compared to joint training and achieves state-of-the-art results on continual learning datasets.