Visual navigation and crop mapping of a phenotyping robot MARS-PhenoBot in simulation

• A workflow integrated visual navigation and field mapping for phenotyping robot. • A multi-camera fusion navigation strategy enabled visual navigation without GNSS. • DL-based segmentation outperformed traditional methods in row extraction. • Multi-object tracking and RTAB-MAP generated 2D crop maps and 3D point clouds. Cultivating high-yield and high-quality crops is important for addressing the growing demand for food and fiber from an increasing population. In selective breeding programs, autonomous robotic systems have shown great potential to replace manual phenotypic trait measurements which are time-consuming and labor-intensive. In this paper, we presented a Robot Operating System (ROS)-based phenotyping robot, MARS (Modular Agricultural Robotic System)-PhenoBot, and demonstrated its visual navigation and field mapping capacities in the Gazebo simulation environment. MARS-PhenoBot was a solar-powered modular robotic platform with a four-wheel steering and four-wheel driving configuration. We developed a navigation strategy that fuses multiple cameras to guide the robot to follow crop rows and transition between them, enabling visual navigation across the entire field without relying on global navigation satellite system (GNSS) signals. Three row-detection algorithms, including thresholding-based, detection-based, and segmentation-based methods, were compared and evaluated in simulated crop fields with discontinuous and continuous crop rows, as well as with and without the presence of weeds. The results demonstrated that the segmentation-based method achieved the lowest average cross-track errors of 2.5 cm for discontinuous scenarios and 0.8 cm for continuous scenarios in row detection. Additionally, a field mapping workflow based on RTAB-MAP (Real-Time Appearance-Based Mapping) and V-SLAM (Visual Simultaneous Localization and Mapping) was developed. The workflow produced the 2D maps identifying crop and weed locations, as well as 3D models represented as point clouds for crop shapes and structures. Using this mapping workflow, the average crop localization error was measured at 6.4 cm, primarily caused by the visual odometry drift. The generated point clouds of crops could support further phenotyping analyses, such as crop height/diameter measurements and leaf counting. The methodology developed in this study could be transferred to real-world robots that are capable of automated robotic phenotyping for in-field crops, providing an effective tool for accelerating selective breeding programs.