Numerical Evaluation and Comparative Analysis of Visual-Inertial SLAM Algorithms ORB-SLAM3, VINS-Fusion, DROID-SLAM, and RTAB-Map

Abstract: Herein, we present a detailed numerical overview and comparison analysis of 4 state-of-the-art visual-inertial SLAM algorithms: ORB-SLAM3, VINS-Fusion, RTAB-Map and DROID-SLAM. For the test, the TUM RGB-D Freiburg1_xyz benchmark dataset (together with its synchronized RGB, depth and ground-truth trajectory data) was used as the base set. We performed the four algorithms on the same set of conditions and did the Umeyama transformation for rigid alignment. We then examined the resulting trajectories in terms of Absolute Trajectory Error (ATE), Relative Pose Error (RPE), drift, and scale consistency. The quantitative results revealed that the performance of the tested methods presented are highly variable between the algorithms from diverse samples. ORB-SLAM3 had the highest localization accuracy with ATE_RMSE 0.0091 m and RPE_RMSE 0.0385 m. VINS-Fusion had the same but near worse results, with an ATE_RMSE= 0.0115 m and RPE_RMSE=0.0458 m. RTAB-Map had a little more than a moderate accuracy (ATE_RMSE=0.0192 m., RPE_RMSE=0.0778 m.) DROID-SLAM experienced the largest errors (ATE_RMSE=0.0365 m, RPE_RMSE=0.1598 m), as well as a large scale distortion (scale ratio=14.99) high in drift (end drift=0.0325 m, segment drift=0.535 m.). The visual (geometric) simulations, through cumulative distribution function (CDFs), per-axis error plots, and radar (radar visual representations), demonstrated that ORB-SLAM3 and VINS-Fusion provided better trajectory stability, lowest drifts, and more uniform scale preservation than the other frameworks. However, DROID-SLAM showed numerous over-scaling and instability for motion estimation. ORB-SLAM3 performance in the final evaluation revealed the best and balanced performance of all tested algorithms. Thus, it is the best for the real-time robotic navigation and mapping applications. For tightly coupled visual-inertial systems, VINS-Fusion is a robust choice.