Sim2Fruit: Towards Reinforcement Learning Training for Sim-to-Real Autonomous Robotic Fruit Harvesting

Harvesting robotic solutions have been shown to significantly improve crop yield, reduce resource consumption, and minimize losses due to pests and diseases. These solutions are built on state-of-the-art agricultural systems that are capable of capturing 3D data of individual crops and constructing digital twins. Prominent issues in the field include extracting meaningful information about the subject plant from the point cloud data and accurately picking fruits in real-world settings. Previously, a deep neural network with PointNet-like architecture was trained to segment plant organelle using 3D point cloud data, allowing for structure identification and monitoring capabilities. However, enabling autonomous plant harvesting remains a challenge. A promising method for robotic systems to address this is sim-to-real transfer learning, where an agent is trained to perform tasks in a realistic simulation environment and deployed in the real-world. In order to enable agricultural tasks such as fruit picking, this work introduces Sim2Fruit, a ROS2 package that integrates plant segmentation algorithms with reinforcement learning (RL) agents to train a robotic arm system in harvesting simulations powered by Gazebo. The results collated evaluate the design, implementation, and effectiveness of the proposed training pipeline using a toy policy network and metrics such as harvest success rate, task completion time, and accuracy of fruit segmentation.