ReplayAR: A Tool for Visual Evaluation of Mixed Reality

In world-locked mixed reality (MR), virtual content is locked in place with respect to the real world. Pose estimation is a key component to create world-locked MR experiences by estimating the device's position and orientation in order to render the virtual content accordingly. Current methods of evaluating world-locked MR include user studies, which are time consuming, and absolute trajectory error (ATE), which does not directly represent what is shown on the user's display. In this work, we propose ReplayAR, a tool that can replay user movement traces and output the corresponding visualizations (renderings) of the MR display. ReplayAR can be used to compare renderings from different MR pose estimation methods side by side, using our proposed Visual Difference metric. We implemented ReplayAR on a Hololens 2 MR headset and used it to evaluate open and closed-source pose estimation methods on standard datasets and our own collected traces. The results suggest that Visual Difference better reflects what is shown on the MR display compared to ATE. We hope that ReplayAR can encourage reproducible evaluation of world-locked MR, and towards this, we release the open-source code.