OccPepSeg-YOLO for Instance Segmentation of Occluded Peppers in Field Images

Agricultural operations such as pepper harvesting, fruit counting, and field phenotyping rely on accurate visual recognition and instance segmentation algorithms. However, pepper fruits in complex field environments often exhibit slender and curved shapes, partial occlusion, ambiguous boundaries, and adhesion between adjacent instances. Existing object detection and instance segmentation methods therefore struggle to obtain complete fruit masks, which adversely affects subsequent fruit counting, contour measurement, and picking-point localization. To improve the instance segmentation accuracy of occluded peppers in complex field scenes, this study proposes OccPepSeg-YOLO, an improved model based on YOLO11n-seg. First, a P2FreqFusion module is introduced to fuse shallow, high-resolution detail features with deep semantic features, thereby enhancing the representation of fruit edges and tip regions. Second, an ASC module is designed to model the directional and scale-related morphological characteristics of pepper fruits, while a BoundaryGate module strengthens responses at occlusion interfaces and boundaries between adjacent instances. Finally, an OccPepSegment multi-scale prototype segmentation head is constructed, and a BDoU loss function is introduced to improve the boundary consistency of instance masks. Experiments on a self-constructed field-pepper instance segmentation dataset showed that OccPepSeg-YOLO achieved M-P, M-R, M-mAP50, and M-mAP50–95 values of 93.87%, 92.09%, 97.17%, and 82.31%, respectively, representing improvements of 5.59, 3.18, 3.83, and 9.52 percentage points over YOLO11n-seg. Further comparisons with representative YOLO-based instance segmentation models, including YOLOv8n-seg, YOLOv9c-seg, YOLO12n-seg, and YOLOv26n-seg, demonstrated that OccPepSeg-YOLO achieved the best overall segmentation performance. In particular, its M-mAP50–95 exceeded the best competing result obtained by YOLOv9c-seg by 8.35 percentage points. Under a unified repeated-inference protocol on an RTX 3090 GPU using FP32 precision, a batch size of 1, and 640 × 640 inputs, OccPepSeg-YOLO achieved a mean inference latency of 15.801 ± 1.238 ms, a P95 latency of 17.323 ms, and a throughput of 63.29 FPS. These results demonstrate that the proposed model can produce more complete pepper instance masks under leaf occlusion, fruit overlap, and complex background conditions, providing technical support for field-pepper recognition, fruit counting, and visual perception by agricultural robots.