Experience: Practical Challenges for Indoor AR Applications

This paper shares the challenges facing today's augmented reality (AR) smartphone applications, particularly in the realm of localization and tracking failure. Our research identifies limitations in current vision-based landmarks such as QR codes and AprilTags, commonly used to aid in localization, and the drawbacks of LiDAR integration in variable lighting conditions, compromising AR's accuracy and functionality. We also examine the constraints of Inertial Measurement Units (IMU) on movement speed, highlighting its impact on the dynamic performance of AR applications. Based on our extensive 316 experimental cases for 113 hours, including 34 case studies with 17 subjects in 2 sites, this paper presents the field with a nuanced analysis of the failure modes inherent in smartphone-based AR localization. We further explore a prototype solution which fuses ultra-wideband (UWB)-based sensing with the vision-based systems to alleviate these failure modes. Our approach addresses the immediate challenges of AR localization and opens avenues for future research and development in creating more spatially aware and interactive digital worlds. All of our demonstration videos, code, and datasets are available here1.