W-ControlUDA: Weather-Controllable Diffusion-assisted Unsupervised Domain Adaptation for Semantic Segmentation

Image generation has emerged as a potent strategy to enrich training data for unsupervised domain adaptation (UDA) of semantic segmentation in adverse weathers due to the scarcity of labelled target domain data. Previous UDA works commonly utilize generative adversarial networks (GANs) to translate images from the source to the target domain to enhance UDA training. However, these GANs, trained from scratch in an unpaired manner, produce sub-optimal image quality and lack multi-weather controllability. Consequently, controllable data generation for diverse weather scenarios remains underexplored. The recent strides in text-to-image diffusion models (DM) enables high fidelity diverse image generation conditioned on semantic labels. However, such DMs must be trained in a paired manner, i.e., image and label pairs, which poses huge challenge to the UDA setting where target domain labels are missing. This work addresses two key questions: <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">What is an optimal approach to train DMs for UDA, and how can the generated data best enhance UDA performance?</i> We introduce W-ControlUDA, a diffusion-assisted framework for UDA segmentation in adverse weather. W-ControlUDA involves two steps: DM training for data augmentation and UDA training using the generated data. Unlike previous unpaired training, our method conditions the DM on target predictions from a pre-trained segmentor, addressing the lack of target labels. We propose UDAControlNet for high-fidelity cross-domain and intra-domain data generation under adverse weathers. In UDA training, a label filtering mechanism is introduced to ensure more reliable results. W-ControlUDA helps UDA achieve a new milestone (72.8 mIoU) on the popular Cityscapes-to-ACDC benchmark and notably improves the model's generalization on 5 other benchmarks.