More Robust and Faster ControlNet with Knowledge Distillation

In the current era of artificial intelligence, models have grown increasingly complex and larger, requiring significant computational resources and storage. This trend motivates research into more efficient model designs, especially methods for reducing model size without sacrificing output quality. This paper proposes a method for reducing the size of ControlNet, a control model for StableDiffusion, for simplicity we used canny for the control images. Our approach involves reducing filter size and block structure while preserving image quality through multilayer Knowledge Distillation (KD), effectively capturing semantic information from control images. We also implemented modulation with Low-Rank Adaptation and Hadamard product (LoHa) for further size reduction. Our experiments demonstrate a 40-80% reduction in model size while preserving the quality of generated images. This research opens new ways to create lighter, still efficient AI models for various applications.