Language-Guided Instance-Aware Domain-Adaptive Panoptic Segmentation

The increasing relevance of panoptic segmentation is tied to the advancements in autonomous driving and AR/VR applications. However, the deployment of such models has been limited due to the expensive nature of dense data an-notation, giving rise to unsupervised domain adaptation (UDA). A key challenge in panoptic UDA is reducing the domain gap between a labeled source and an unlabeled target domain while harmonizing the subtasks of seman-tic and instance segmentation to limit catastrophic inter-ference. While considerable progress has been achieved, existing approaches mainly focus on the adaptation of se-mantic segmentation. In this work, we focus on incorpo-rating instance-level adaptation via a novel instance-aware cross-domain mixing strategy IMix. IMix significantly en-hances the panoptic quality by improving instance segmen-tation performance. Specifically, we propose inserting high-confidence predicted instances from the target domain onto source images, retaining the exhaustiveness of the resulting pseudo-labels while reducing the injected confirmation bias. Nevertheless, such an enhancement comes at the cost of degraded semantic performance, attributed to catastrophic forgetting. To mitigate this issue, we regu-larize our semantic branch by employing CLIP-based do-main alignment (CDA), exploiting the domain-robustness of natural language prompts. Finally, we present an end-to-end model incorporating these two mechanisms called LIDAPS, achieving state-of-the-art results on all popular panoptic UDA benchmarks. https://github.com/elhamAm/LIDAPS