DMFFT: improving the generation quality of diffusion models using fast Fourier transform

In this study, we demonstrate the significant development potential of diffusion U-Net extraction features transferred to the frequency domain, opening up a new perspective for diffusion models and generating a new optimization idea for diffusion model-related research. The generating quality of Text-to-Image (T2I) or Text-to-Video (T2V) can be significantly enhanced by modifying the key indicators of U-Net frequency domain features. We first investigated the two types of modules in the sampling process, CrossAttnUpBlock and UpBlock, on U-Net, and then examined the effect of fine-tuning modules on U-Net feature extraction from backbone and lateral skip connections. Finally, it is decided to modify the feature extraction procedure of the CrossAttnUpBlock as the entrance to increase the overall diffusion generation quality. At the same time, employing the traditional Fourier transform method in image processing, the influence of frequency domain elements such as frequency, amplitude, and phase on image generation is investigated in the upsampling CrossAttnUpBlock of diffusion models, resulting in the method DMFFT, which can improve T2I or T2V quality without training or fine-tuning. During the experiments, the scaling factors for high frequency, low frequency, amplitude, and phase are adapted, generating a variety of characteristic performance results. A large number of experiments show that the proposed method, DMFFT, can significantly improve the semantic alignment, structural layout, color texture, and temporal consistency of images or videos generated by baseline models, as well as improving the artistry and diversity of the generation effect.