SAFF-DETR: An End-to-End Object Detection Network for Remote Sensing Images With Targets of Varying Sizes Based on Scale Adaptation and Frequency Fusion

Deep learning-based object detection algorithms have achieved significant success in the field of computer vision. However, the wide range of target sizes in remote sensing images poses a challenge for single algorithms to detect objects of varying sizes effectively. To address this issue, this paper proposes an end-to-end object detection algorithm for remote sensing images based on Scale Adaptive and Frequency Fusion DETR (SAFF-DETR), which designs a frequency feature enhancement and fusion mechanism to handle targets of varying sizes within a single framework. First, in order to improve the Transformer-based detectors’ ability to small targets, a multibranch representation fusion (MRF) module is proposed to fuse shallow layer frequency representations, boosting the network’s ability to perceive small targets. Furthermore, Cross-layer Spatial and Channel Frequency Attention (CSFA and CCFA) is designed to enable efficient frequency feature interaction across multi-scale features, enhancing the representation capability for targets of different sizes. Moreover, by integrating the two aforementioned attention mechanisms, the Cross-layer Channel-Spatial-wise Frequency Fusion (CCSFF) structure is introduced to realize global feature interactions in one step without repetitive up and down-sampling operations, by which patch division-based Transformer architecture is designed to enhance scale adaptability for object detection. Experimental results over several benchmark datasets demonstrate that the proposed SAFF-DETR can handle extremely varying-size targets and outperforms several SOTA algorithms. The source code will be available at https://github.com/Jianglin-Zhao/SAFF-DETR.