Visual-LiDAR 3D semantic mapping for the unconstructed, large-scale field environment

Most mobile manipulation tasks in outdoor field environments, such as weeding, fruit picking, and pesticide spraying, demand a 3D semantic environmental model that goes beyond the classical attributes of geometry and appearance to support task planning and execution. However, mapping such an unconstructed environment remains challenging because the environmental conditions are prone to sensor degradation issues to LiDAR or camera-based methods in terms of semantic perception and odometry estimation. To tackle the calibration-dependent issue that exists in association-by-projection methods, a distance-centered seed growth-based method for vision-aided point cloud segmentation is presented, in which the semantic labels automatically expand among the point cloud by exploiting 3D geometrical characteristics. Furthermore, based on such a 3D semantic perception module, a 3D semantic mapping system for large-scale, dynamic scenes is implemented. Experimental evaluation in the real world is conducted. The results suggest that the proposed method is able to overcome the dynamic interference and the unconstructed conditions and to achieve semantic segmentation, pose estimation, and mapping in real-time.