Synergizing natural visual features and 3D building models for robust indoor localization in mixed reality environments

The rapidly developing Mixed Reality (MR) technology provides a powerful visualization and interaction approach for outdoor and indoor environments. A main challenge of MR is to accurately align (register) virtual information or objects to the real world, which is a prerequisite for a good visualization and immersive experience. This paper introduces a visual localization method for MR based on the natural feature door and as-planned Building Information Modeling (BIM) models as reference data. The proposed method incorporates a YOLOv5-assisted and AprilTag-inspired 2D door corner detector as well as an interactive extraction approach for 3D door corners from BIM models. A Perspective-n-Point (PnP) method is applied to estimate the pose of the MR device. The proposed method was deployed and evaluated on the Microsoft HoloLens 2. The results indicate that as-planned BIM models and natural visual features such as doors prove to be an effective approach to achieve state-of-the-art localization accuracy in indoor environments.