DIFFUSE2ADAPT: Controlled Diffusion for Synthetic-to-Real Domain Adaptation

Synthetic data generated from graphics engines has been shown to be effective for learning, while also being a cost-effective alternative to annotating real-world data. However, models trained on synthetic data often suffer from performance degradation when applied to real-world data due to the domain gap. In this paper, we propose Diffuse2Adapt, a novel unsupervised domain adaptation (UDA) approach that leverages controlled diffusion models to bridge the synthetic-to-real domain gap. Our method utilizes text-to-image generative models to translate synthetic images to the target domain while preserving class semantics. We introduce two methods to reduce the domain gap: (1) incorporating target domain context extracted from multimodal language models, and (2) capturing target domain style via learned textual tokens. Extensive experiments are performed on three synthetic-to-real domain adaptation benchmarks, VisDA-2017, S2RDA-49, and S2RDA-MS-39. Diffuse2Adapt outperforms state-of-the-art methods by +3.00% on VisDA-2017, +2.55% on S2RDA-49 and +1.01% on S2RDA-MS-39. Code will be released.