GMG-LDefmamba-YOLO: An Improved YOLOv11 Algorithm Based on Gear-Shaped Convolution and a Linear-Deformable Mamba Model for Small Object Detection in UAV Images

Object detection plays a crucial role in remote sensing and UAV image technology, but it faces the challenge of speed and accuracy in multi-scale dense small target mission detection scenarios and is susceptible to noise interference, such as weather conditions, lighting changes, and occluded backgrounds in complex backgrounds. In recent years, Mamba-based methods have become hot in the field of object detection, showing great potential in capturing remote dependencies with linear complexity but lacking deep customization of remote sensing targets. Based on this, we propose GMG-LDefmamba-YOLO, which contains two core modules: the Gaussian mask gear convolution module forms a gear-shaped receptive field through improved convolutional splicing to enhance the extraction of small target features and combines the Gaussian mask mechanism to dynamically modulate the feature weights to suppress complex background interference. The linear deformable Mamba module integrates linear deformable sampling, a spatial state dual model, and residual gating MLP components, integrating the advantages of flexible capture of local features and efficient modeling of global dependence, dynamically adapting to target scale changes and spatial distribution, and reducing computational costs. Experiments on DOTA-v1.0, VEDAI, and USOD datasets show that the mAP50 of the model reaches 70.91%, 77.94%, and 90.28%, respectively, which is better than the baseline and mainstream methods, and maintains the lightweight characteristics, providing efficient technical support for remote sensing monitoring, UAV inspection, and other fields.