Improved YOLOv8 for multi-colored apple fruit instance segmentation and 3D localization

Robotic apple harvesting requires precise instance segmentation and 3D localization, especially for multi-colored apples under complex orchard conditions with occlusions and variable lighting. Current deep learning methods lack robustness and accuracy for such scenarios, limiting automation. This study proposes improved YOLOv8-based models to enhance segmentation of multi-colored apples, combined with a high-precision 3D localization pipeline to advance practical robotic harvesting. To address these issues, this study collected apple images in three colors from two locations, creating a dataset of 5171 images. Four enhanced YOLOv8-based models—RA-YOLO, GA-YOLO, YA-YOLO, and MCA-YOLO—were proposed for segmenting red, green, yellow, and mixed multi-colored apples. RA-YOLO integrates the GD mechanism and EMBConv structure based on EfficientNet's MBConv. GA-YOLO replaces standard convolutions with dynamic serpentine convolution and adds the P6 layer for large object detection. YA-YOLO utilizes deformable convolution (DCNv2) and introduces the new attention mechanism MPCA. MCA-YOLO combines the P6 layer, DCNv2, and EMBConv structure, merging the strengths of other models. RA-YOLO, GA-YOLO, and YA-YOLO achieved mAP values of 95.2 %, 96.4 %, and 95.4 %, respectively, for single-colored apple instance segmentation, surpassing baseline models and those in existing literature. MCA-YOLO achieved mAP values of 95.6 %, 96.6 %, and 94.6 % for single-colored apples and 95.6 % for mixed multi-colored apples. Ablation experiments validated the necessity of each module. Finally, a high-precision 3D localization and shaping pipeline was developed, achieving an average localization error of 2.636 mm and a shaping error of 0.768 mm, enabling millimeter-level localization and sub-millimeter-level shaping for apple harvesting optimization.