RPGO-MR: Relocalization Pose Graph Optimization for accurate and latency-aware Mixed Reality localization

Mixed Reality (MR) is becoming increasingly adopted for Human–Machine Collaboration (HMC), particularly in applications like 3D visualizations, where accurate spatial transformations are critical for integrating pre-built map data with MR headsets. A key component of this integration is relocalization, which estimates the relative camera pose within a pre-built map frame to align the reference frames of MR environments and external maps. However, existing MR relocalization methods face several practical challenges, including unstable single-frame relocalization, accumulated VIO drift, delayed pose updates, and limited access to proprietary SLAM/VIO systems in commercial devices. To overcome these challenges, we introduce Relocalization Pose Graph Optimization for MR headsets ( RPGO-MR ), a plug-in external framework designed to enhance real-time relocalization accuracy and stability. RPGO-MR introduces a novel Plane-based Single-Shot image reLocalization (PSSL) method, which extends a conventional image-retrieval and feature-matching pipeline by using plane-based filtering to improve 2D–3D correspondence reliability. The resulting relocalization poses, headset VIO relative poses, and map-based visual constraints are then fused in a sliding-window factor graph. Delayed relocalization measurements are inserted into their corresponding historical pose nodes, while real-time pose output is maintained by propagating the latest optimized pose using high-frequency VIO updates. Importantly, the core contribution of RPGO-MR is its external plug-in fusion framework, which operates on commercial MR headsets without requiring access to their proprietary SLAM or VIO internals. RPGO-MR reduced image-plane reprojection error relative to the corresponding raw relocalization baseline by 16.28%, 16.3%, and 12.65% in the indoor, small-scale outdoor, and large-scale outdoor experiments, respectively. While image relocalization requires approximately 2.16 s per query, the RPGO-MR requires only approximately 0.15 s per update, enabling delayed relocalization measurements to be incorporated while maintaining responsive pose output through high-frequency VIO propagation. Our implementation will be released at https://github.com/cviss-lab/RPGO-MR .