Enhanced machine vision system for field-based detection of pickable strawberries: Integrating an advanced two-step deep learning model merging improved YOLOv8 and YOLOv5-cls

In order to successfully deploy robotic harvesting in open field conditions, the development of an effective machine vision system becomes crucial. In this research, we proposed a novel two-step deep learning model consisting of a modified YOLOv8s and a YOLOv5s-cls to accomplish strawberry detection and pickability classification (whether a mature fruit is pickable by a robot). Firstly, the YOLOv8s was enhanced by incorporating C3x modules and an additional head network structure, specifically tailored for accurate strawberry detection. To further improve training performance, the α -IOU (intersection over union) technique was integrated. Subsequently, the YOLOv5s-cls was utilized to determine suitability of the detected mature strawberries. Through evaluations, Model D (+C3x+head+ α IoU), which was a model based on modifying YOLOv8 using the new modules and techniques mentioned above, was found to perform the best among the tested models achieving the highest AP scores of 84.2% in Stage I (immature), 77.8% in Stage II (nearly mature), and 87.8% in Stage III (mature), along with the highest mAP of 83.2%. Overall, this modified model achieved a 2.5% improvement in mAP compared to the same achieved by original YOLOv8s model. Despite a slightly slower inference speed of 8.4 ms per image, Model D maintains real-time capabilities, making it an optimal choice for strawberry detection. Additionally, YOLOv5s-cls was identified as the preferred model for classifying mature strawberries into pickable and unpickable groups, offering a good inference speed of 2.8 ms per image and comparable accuracy with other compared models including YOLOv8s-cls, ResNet 18, EfficientNet-b0, and EfficientNet-b1. Finally, the combined two-step model developed in this study was evaluated in 10 different field scenarios from a completely different strawberry field that was not used in model training and initial testing. In this validation test the machine vision system achieved an AP of 89.0%, 82.0%, and 90.0% in detecting strawberries from Stage I, II, and III while the classification accuracy was 100.0% in unpickable group and 95.0% in pickable group. The results showed that the developed two-step machine vision system has a potential to improve the overall robotic harvesting system for strawberries grown in open-field conditions. • Developed a machine vision approach for strawberry harvesting combining fruit detection and occlusion classification models. • Modified and enhanced YOLOv8 for strawberry detection in open-field environments. • Introduced a classification model for accurate pickability assessment. • Conducted field evaluation of the proposed method.