From Static Dense to Dynamic Sparse: Vision-Radar Fusion-Based UAV Detection

Precise unmanned aerial vehicle (UAV) detection over long distances is of crucial importance for guaranteeing the airspace security. Although deep learning-based vision detectors have been developed, they still rely on a large amount of hand-crafted fixed feature priors. The existing static dense-based detectors suffer from the severe mismatch and imbalance between the small size and the high mobility of UAVs. To solve the problem, a novel multimodal fusion-based dynamic sparse UAV detection framework is proposed. The framework reformulates the feature priors in a completely dynamic sparse paradigm by using the radar data. Based on the framework, a vision-radar fusion-based dynamic sparse network (Vira-DSNet) is proposed for more balanced and robust UAV detection. The Vira-DSNet exploits our designed dynamic sparse candidate generator and radar-guided semantic feature transform to generate a small set of customized high-quality object candidates and semantic features based on the radar data. Moreover, based on Hungarian bisection matching, our Vira-DSNet eliminates the post-processing and is completely end-to-end differentiable. Furthermore, the Vira-DSNet is deployed in our developed actual vision-radar fusionbased UAV detection system to evaluate the performance in the practical applications. Experimental results demonstrate that our Vira-DSNet achieves an average precision AP<sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">50</sub> of 88.2%. It is also shown that the average recall AR<sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sub> of Vira-DSNet is higher than the state-of-the-art scheme by 10.1%, while maintaining the real-time performance.