DFL-YOLO: Density-Guided Frequency Decoupling and Large Selective Kernels improve dense multi-class tree crown detection

Multi-class tree crown detection in unmanned aerial vehicle (UAV) imagery is essential for forest monitoring but remains challenged by severe target-scale variation, feature ambiguity caused by crown overlap and background interference, and unstable bounding box regression in crowded scenes. To address these coupled errors, we propose Density-Guided YOLO with Frequency Decoupling and Large Selective Kernels (DFL-YOLO), a YOLOv11-based framework that combines scale-context construction, density-conditioned feature disambiguation, and improved bounding box regression within a progressive detection process. First, structurally pruned Large Selective Kernel (LSK) modules combine cascaded depthwise convolutions with spatial selection to enrich local and long-range context along selected backbone and neck paths. Second, our Density-Guided Frequency Decoupling Branch (DFDB) predicts a Gaussian-supervised density mask from the resulting context-enhanced P4 feature and applies complementary modulation to its high- and low-frequency components, preserving boundary-sensitive responses while suppressing background interference. The modulated P4 feature is then fused with the P3 and P5 paths and passed to the multiscale detection head. Finally, Multi-Strategy Weighted Intersection over Union (MW-IoU) supervises bounding box regression by combining MPDIoU geometric constraints, the non-monotonic focusing mechanism of Wise-IoU, and task-oriented soft weights. On the Mixed NEON dataset constructed from NEON and Roboflow imagery, DFL-YOLO achieved 82.2% mAP@50 and 54.7% mAP@50:95, representing absolute improvements of 2.6% and 4.2% over the YOLOv11l baseline, respectively. After dataset-specific retraining on VisDrone2019, DFL-YOLO achieved 54.5% mAP@50 and 33.7% mAP@50:95, showing that the architecture can be trained for another dense aerial-imaging domain. In general, DFL-YOLO is a reliable method for detecting multi-class tree crowns in intricate forest environments.