Evaluating ORB-SLAM 3 Performance Using a Photogrammetry-Based Reference Trajectory

Abstract. The robust evaluation of Visual Simultaneous Localization and Mapping (vSLAM) systems is fundamental to their development and deployment. However, this process is often constrained by the reliance on expensive and complex external infrastructure, such as laser trackers or motion capture systems, to provide accurate ground-truth trajectories. This paper introduces a novel and self-contained methodology for the high-fidelity evaluation of stereo vSLAM and stereo-inertial algorithms. Our approach leverages the very same image sequence used by the SLAM algorithm to generate a dense, globally optimized photogrammetric model. The proposed methodology comprises two fundamental steps, the first step consists of validating photogrammetry as a ground truth method. For this purpose, the linear displacement measured by photogrammetry was compared with the displacement of a precision guide, which was benchmarked against a laser interferometer as the standard. Once the reference was validated, the second step assessed the performance of ORB-SLAM 3 on a free trajectory within a complex environment, by directly comparing the SLAM result to the trajectory generated by photogrammetry. The accuracy was then quantified using standard metrics, including Absolute Trajectory Error (APE) and Relative Pose Error (RPE). The results validate our approach as an accessible, low-cost, and reliable alternative for benchmarking vSLAM systems, enabling a way of performance analysis using only the data from the sensor suite under evaluation.