Robust Real-World Image Dehazing via Knowledge Guided Conditional Diffusion Model Finetuning

Due to the domain gap, the dehazing models trained from the synthetic images suffer poor generalization performance on real-world images. To address this issue, we pro-pose a Knowledge guided Conditional Diffusion (KCDiff) model finetuning method, which enables both the domain knowledge adaptation from the synthetic images and general knowledge guidance from the real-world images. More specifically, our KCDiff comprises two modules, i.e., the Conditional Image Generation (CIG) and Dehazing Instruction Generation (DIG). For CIG, we freeze a pre-trained latent diffusion model, add learnable conditioning control layers with Low-Rank Adaptation (LoRA) blocks, and include skip connections with zero-initialized convolutional layers, all of which play a fundamental role in image dehazing. Meanwhile, the DIG utilizes a large vision-language model LLaVA to extract the semantic content of the input hazy image and redescribe it in clear weather, which serves as the control instruction of CIG. To mitigate potential artifacts in CIG caused by misinterpretation of DIG's instructions, we further enforce depth and physical model-based reconstruction consistency constraints on both dehazing and hazy images. In the training phase, CIG is trained with the paired synthetic images to adapt the diffusion prior to the domain knowledge of image dehazing and finetuned with unpaired real-world images to suppress the domain gap with general knowledge guidance from the atmospheric scattering and depth perception. Experiments on real-world databases demonstrate the superiority of the proposed method over many state-of-the-art image dehazing models.