Neighboring-aware optimization for end-to-end detection transformer

The Transformer-based DETR framework has achieved end-to-end set prediction, driving technological innovation in the field of object detection. However, it has two key limitations: First, the cost function overlooks semantic similarity between queries and targets, and the Hungarian algorithm fails to capture their intrinsic relationships, resulting in inaccurate matches. Second, the training phase imposes positional constraints only on positive samples and lacks an exclusion mechanism for neighboring samples, limiting detection performance. This paper analyzes the spatial proximity and semantic similarity between neighboring negative samples and positive samples, which weakens the model's ability to distinguish between positive and negative samples. To address this, a Neighboring-Aware Optimization Strategy (NAOS) is proposed and applied to the DETR framework, resulting in the NAO-DETR method. The proposed method enhances the Hungarian matching process by incorporating a semantic similarity cost between queries and targets, thereby alleviating query–target matching ambiguity. In addition, it integrates explicit supervision through geometric and coverage constraints on neighboring samples, which effectively suppresses positional interference during regression and consequently improves the accuracy of object localization. Experimental results demonstrate that NAO-DETR achieves superior performance, surpassing baseline models DINO and Deformable-DETR by 2.1% and 5.7% in mAP on COCO datasets, respectively. Moreover, NAOS demonstrates strong generalization, effectively transferring to DETR framework to improve its detection performance.