Development of a pumpkin fruits pick-and-place robot using an RGB-D camera and a YOLO based object detection AI model

• The detail of each module of the new developed pumpkin harvesting robot was illustrated in the paper. • The fruits detection results of using YOLO v2 to YOLO v8 have done and the results are compared. • A one-shot 3D camera and robot calibration method was proposed. It is a hard job for farmers to harvest heavy fruits such as pumpkin fruits because of the aging problem of farmers. To solve this problem, this study aims to develop an automatic pick-and-place robot system that alleviates labor demands in pumpkin harvesting. We proposed a system capable of detecting pumpkins in the field and obtaining their three-dimensional (3D) coordinate values using artificial intelligence (AI) object detection methods and RGB-D camera, respectively. The harvesting system incorporates a crawler-type vehicle as the base platform, while a collaborative robot arm is employed to lift the pumpkin fruits. A newly designed robot hand, mounted at the end of the robot arm, is responsible for grasping the pumpkins. In this paper, we utilized various versions of YOLO (from version 2 to 8) for pumpkin fruit detection, and compare the results obtained from these different versions. The RGB-D camera, that was mounted at the root of the robot arm, captures the position of the pumpkin fruits in camera coordinates. We proposed a calibration method can simply transform the position to the coordinates of robot arm. In addition, we finished all the software and hardware of the pumpkin fruits pick-and-place robot system. Field experiments were conducted at an outdoor pumpkin field. The experiments demonstrate the fruits detection accuracy rate exceeding 99% and a picking success rate surpassing 90%. However, fruits that were surrounded by excessive vines could not be successfully grasped.