Air‐to‐Ground Target Detection and Tracking Based on Dual‐Stream Fusion of Unmanned Aerial Vehicle

ABSTRACT Both visible and infrared images are important sources of intelligence information on the battlefield, and air‐to‐ground reconnaissance by UAV is an important means to obtain intelligence. However, there are great challenges in ground target detection and tracking, especially in complex battlefield environments. Aiming at the problem of insufficient accuracy of target detection by a single type of sensor in the battlefield environment at this stage, a target detection method by fusion of visible and infrared images is proposed in this paper, which is called ReconnaissanceFusion‐YOLO (RF‐YOLO), and with the help of infrared imagery, it can effectively improve the accuracy of target detection in the case of insufficient light. The performance of target detection in the battlefield is significantly improved by introducing two key innovative modules: dual feature fusion (DFF) module and feature fusion corrector (FFC) module. The DFF module enhances multi‐channel feature fusion through a novel concatenation and channel‐wise attention mechanism, while the FFC module performs feature correction between parallel streams using spatial and channel‐wise weights, addressing noise and uncertainty in different modalities. These modules are integrated on top of a dual‐stream YOLO architecture, allowing for effective fusion of visible and infrared information. RF‐YOLO was trained and evaluated using the FLIR data set, containing 5142 pairs of strictly aligned visible and infrared images. Results demonstrate that RF‐YOLO significantly outperforms benchmark networks in terms of robustness requirements. Specifically, the large model of RF‐YOLO achieves an mAP of 0.831, which is a significant improvement compared to the YOLOv5l inf benchmark's 0.739. This represents an improvement of over 12% in detection accuracy. Additionally, RF‐YOLO offers a Nano version that balances accuracy and speed. The Nano version achieves an mAP of 0.765, while maintaining a model size of only 11.5 MB, making it suitable for deployment on UAV edge computing devices with limited resources. To validate the practical applicability of our approach, this paper successfully implements target detection and tracking on a real UAV's edge computing device using the ROS system and SiameseRPN, combined with the proposed RF‐YOLO. Real‐world flight tests were conducted on an internal playground, demonstrating the effectiveness of our method in actual UAV applications. The system achieved a processing rate of approximately 10 fps at 640 × 640 resolution on an NVIDIA TX2 edge computing device, showcasing its real‐time performance capability in practical scenarios. This study contributes to enhancing UAV‐based battlefield reconnaissance capabilities by improving the accuracy and robustness of target detection and tracking in complex environments. The proposed RF‐YOLO method, along with its successful implementation on a real UAV platform, provides a promising solution for advanced military intelligence gathering and decision‐making support.