MRescue: A mixed reality system for real-time navigation and rescue in complex multi-floor buildings

In high-rise buildings, complex layouts, and frequent structural modifications can make emergency rescues challenging. Conventional 2D rescue plans offer no real-time guidance and can be difficult for rescuers to interpret under stress. To overcome these limitations, we introduce an advanced Mixed Reality (MR) application designed for real-time rescue assistance in multi-floor buildings. Built for the Meta Quest 3, a cost-efficient standalone MR headset, our system enables users to scan, update, and navigate a dynamic 3D model of their surroundings. We implement external data storage and utilize spatial anchors to ensure accurate realignment to bypass the Meta Quest 3’s constraint of storing only 15 rooms. Additionally, the application utilizes the A* algorithm to dynamically calculate optimal routes based on the user’s real-time location, taking into account the room layout and any obstacles inside. Users can navigate using either a floating 3D minimap or a 2D minimap anchored to their left hand, with staircases seamlessly incorporated into navigation routes, including virtual door warnings and automatic removal of navigation cues near stairs for safety. To improve user experience, we have implemented hand tracking for interactions. We conducted a study with a large sample of participants to evaluate usability and effectiveness. This study included the NASA Task Load Index (NASA-TLX) to assess the cognitive load, along with the System Usability Scale (SUS), the Self-Assessment Manikin (SAM), the Single Ease Question (SEQ), task completion times, and a post-evaluation feedback questionnaire. The results demonstrate that the system achieves high usability, low cognitive workload, and positive user experience while supporting situational awareness, user confidence, and efficient navigation. The results indicate that a system based on an MR headset has the potential to improve situational awareness and decision-making in dynamic indoor environments. • The MR-based system provides real-time guidance within multi-floor build- ings. • Hand tracking replaces controllers for intuitive interaction and improved accessibility. • A* algorithm enhances pathfinding by considering real-world obstacles. • Dynamic 3D model updates ensure accurate navigation despite structural changes. • The user study evaluates the system’s usability, effectiveness, cognitive load, and task completion time.