DA-RTDETR: domain-adaptive RT-DETR with feature fusion and category-level constraints

Transformer-based object detection has attracted significant attention recently due to its promising performance. As the first work of DETR-like algorithms in the field of real time object detection, Real-Time Detection Transformer (RT-DETR) has achieved excellent performance on COCO val2017 and reached a new state of the art. However, when applied to new scenarios with significantly different imaging conditions and no annotated data, it often encounters challenges. To alleviate the issue of domain shift, this paper designs DA-RTDETR, an unsupervised domain adaptive RT-DETR method. DA-RTDETR consists of Feature Fusion Enhanced Discriminator (FFED) and Nonlinear Feature Constraint on category-level tokens (NFC). Specifically, FFED aligns multi-scale features from the backbone and encoder through hierarchical feature adaptation (HFA) and feature fusion (FF), enhancing the domain discrimination ability of the backbone and encoder. NFC extends the Deep CORAL loss to the category-level tokens of the decoder to minimize the difference in second-order statistics between the source and target domain. Our experiments on two challenging benchmarks demonstrate the effectiveness of our method, with a particularly notable relative improvement of 25.3 $$\%$$ over the baseline on the Foggy Cityscapes dataset. Code is released at https://github.com/GodFish520/DA-RTDETR .