Structure-Aware Stable Diffusion

Latent diffusion models (LDMs) have long been the state of the art for image inpainting, class-conditional image synthesis and various generative vision tasks, such as unconditional and text-conditional image generation, super-resolution, etc., while significantly reducing the compute requirements compared to the primitive Denoising Diffusion Implicit Models (DDIMs). Although latent diffusion models have significantly accelerated the training speed of diffusion models through their latent-based architecture and substantially reduced computational requirements, the training process remains time-intensive and generates a considerable carbon footprint. This raised the growing requirement for a viable alternative that can train faster while maintaining training stability and improving sample generation quality! In this paper, we introduce Structure-Aware Stable Diffusion (SASD), which employs a joint guidance technique using text and SSIM-based distortion conditioning to make the U-Net in the reverse process of the Latent Diffusion Model (LDM) architecture converge faster while maintaining training stability, hence effectively speeding up the model training and also reaching better sample generation quality with an average improvement of ∼14% in FID at comparatively lesser epochs than the original model architecture of LDMs.