Development of a grape-harvesting robot using a multi-step detection method based on AI and a position-estimation algorithm

The consumption of Japan wine is increasing year by year, and the country's grape-cultivation industry is facing many challenges. Grape harvesting, as a labor-intensive industry, requires a large amount of labor input, and many problems (e.g., the aging population and labor shortage) have already had huge impacts on grape cultivation, making the need for automated grape harvesting increasingly urgent. The fast and accurate identification and positioning of crops is an essential ability for crop-harvesting robots. To achieve this goal, we propose a two-step vision recognition method that combines the YOLOv7 and YOLOv7-Pose algorithms which helps a harvesting robot quickly and accurately complete grape harvesting in actual field environments. A color camera is used first to capture images in a vineyard, and then the YOLOv7 model is used to quickly identify and locate the position of the grapes in the captured images. The images are cropped according to the coordinates of the detected bounding boxes in order to simulate the camera approaching the grapes. When only one or a small amount of target grape clusters exist in the image, the YOLOv7-Pose model is used to detect the key points of the grapes and output their coordinates to predict the correct cutting point for the robot's hand. We also conducted experiments to test the efficacy of the designed method. Datasets were created for model training and testing, coupled with the use of reasonable indicators for evaluation. The testing results successfully affirmed the applicability and feasibility of the designed method. The insights and methodologies elucidated in this study can serve as valuable references for researchers delving into the realms of crop identification and harvesting.