Dynamic Semantic SLAM Based on Panoramic Camera and LiDAR Fusion for Autonomous Driving

This paper proposes a semantic SLAM methodology for autonomous driving, integrating panoramic camera and LiDAR fusion to address pose estimation inaccuracies and lack of semantic information in dynamic environments. On the LiDAR side, panoramic semantic segmentation results are projected onto point clouds and combined with clustering for instance segmentation, classifying points into unknown-motion landmarks, static landmarks, and ground points. Precise pose estimation is then achieved using purely static and semi-static landmarks. The visual component incorporates data augmentation and instance segmentation of front-view images, leveraging SuperPoint for feature extraction. 3D dynamic landmarks are projected onto segmentation results to generate dynamic-static masks, enabling the elimination of feature points within dynamic regions. A robust visual odometry system is implemented through the ORB-SLAM2 tracking framework, enhanced by SuperGlue feature association within a multi-level tracking mechanism. At the fusion level, a decision-layer fusion algorithm integrates both odometry sources. Furthermore, pose optimization is achieved through dual loop closure detection with inputs from both LiDAR and visual sources. The proposed system was validated using the CQU01-CQU04 datasets collected on a UGV, with high-precision GNSS serving as the ground truth. Experimental results demonstrate that the system achieves mean of average error of estimated positions, standard deviation of estimated position errors, and position error at the trajectory end of 0.446m, 0.236m, and 0.775m, respectively. With loop closure enabled, the performance metrics showed enhancement to 0.320m, 0.181m, and 0.617m. This research demonstrates that our method improves localization accuracy while constructing semantic map comprising purely static elements.