RamSeg: Real-time Multimodal Semantic Segmentation for Autonomous SLAM Systems

Semantic segmentation significantly enhances the capabilities of autonomous SLAM systems. We proposed RamSeg, a multimodal 3D semantic segmentation approach for autonomous SLAM systems, supporting the real-time transmission and storage of segmentation results with localization information. The camera images and LiDAR point clouds are acquired simultaneously, then image segmentation is performed. In the point to image projection module, the points within the camera's view frustum are projected onto the image segmentation mask and the corresponding label is assigned to the point to obtain 3D segmentation result. Ultimately, the 3D semantic segmentation map of the entire scene is obtained by merging the data from each moment utilizing SLAM. The experiment was carried out around the Star Lake Building on Wuhan University using the Luo-Jia Explorer system. The experimental results indicate that the approach can reach a maximum inference speed of 14Hz with RTX 4070 graphic card while the data captured by the system is 10Hz, meeting the real-time requirement while having excellent segmentation result in indoor and outdoor scenes.