Text-to-image with frequency domain diffusion models

Diffusion Models have shown excellent performance in Image Generation. However, existing Diffusion Models usually over-focus on the spatial domain and ignore the information in the frequency domain, which results in the edge of the generated image is not clear enough. To solve the problem, we propose frequency domain diffusion models (FDDMs). FDDMs effectively leverage both spatial domain and Fourier frequency domain information throughout the training phase. First, we attenuate the low-frequency information from the frequency domain image. Second, we obtain the partial weights of the Reverse Diffusion U-network (Unet) through a frequency domain feature extractor and a weight generation network. Finally, we linearly interpolate these weights with the original weights of Unet and realize the introduction of frequency domain information in image generation. We conducted extensive experiments under the dataset Oxford-102 Flower in the task of text-to-image generation. The experimental results demonstrate that the images generated by FDDMs exhibit significantly sharper edges compared to those produced by alternative diffusion models. Besides, FDDMs establish state-of-the-art performance in the field of text-to-image generation, reducing the average FID score by 5.6%, increasing the IS score by 1.0%, and increasing the CLIP score by 3.6%. The project website can be found here: https://github.com/LantuXu/FDDM.