Modified YOLOv8 for Real-Time Multi-Class Uncooperative Aerial Vehicle Detection and Classification

With the rapid increase in UAVs (fixed-wing or rotor-based) for commercial and recreational applications, serious concerns arise regarding their misuse. Moreover, their small size complicates detection, distinguishing them from other aerial vehicles like airplanes, helicopters, and birds. This paper introduces an efficient model for UAV detection and classification from a distance using visual sensors. We enhance the YOLOv8 model by integrating a novel attention module called Multi Group Dual Attention Module (MGDAM) for small-object detection. Additionally, a synthetic dataset is generated for multiple classes (drones, airplanes, helicopters, and birds) that closely resemble real-world camera-captured images. Unlike many studies focusing only on mAP@50, this paper emphasizes mAP50–95 to evaluate performance in detecting tiny objects. To identify an optimal configuration, thorough ablation experiments were conducted involving different attention modules— Convolutional Block Attention Module (CBAM), Coordinate Attention, Shuffle Attention, Squeeze-And-Excitation, Efficient Channel Attention (ECA), and proposed MGDAM. Experimental results show that the MGDAM-enhanced YOLOv8 model achieves a 0.4% gain in mAP50 and a 1.7% boost in mAP50– 95 over the YOLOv8 baseline. Finally, real-world testing confirms robust UAV detection under challenging conditions (e.g., extreme lighting and clutter), demonstrating the method’s efficacy in improving aerial surveillance and security measures.