YOLOv8-WRTNet: Instance Segmentation for Aerial Images under Weak and Repetitive Texture Conditions

Aiming at the problem of insufficient accuracy of image detection and segmentation of traditional YOLOv8 methods in weak texture environments, this paper proposes an improved YOLOv8 method, YOLOv8-WRTNet, to enhance the accuracy and robustness of target detection segmentation. The algorithm firstly enhances the diversity and stability of feature expression by introducing an improved version of C2f module, which utilises feature separation, convolution processing and residual joining. Secondly, the BiFormer module is adopted to strengthen the model’s ability to capture and process multi-level features through dual-path feature fusion and multi-scale attention mechanism, which particularly enhances the target recognition ability in complex repetitive texture environments. Subsequently, combined with the FocalNext module, the model’s target detection and segmentation ability in complex backgrounds is enhanced through the integration of inflated depth convolution and jump connection, which effectively fuses fine-grained local information with coarse-grained global information. Finally, a custom Wise IoU loss function is designed and introduced to further improve the detection robustness of the model in complex scenes by dynamically adjusting the matching accuracy of the model to the target boundary. The experimental results show that the improved YOLOv8 method improves both accuracy and performance in the detection and segmentation tasks dealing with weak texture environments and complex backgrounds.