Positional Tracking of Physical Objects in an Augmented Reality Environment Using Neuromorphic Vision Sensors

Immersive systems such as Augmented Reality (AR) devices are being integrated across multiple domains, from manufacturing to health, environment and education. The Industry 5.0 focus on human-centric interfaces to these systems and the desire to enable them for enhancing the skill-sets of the user, present new challenges. One of these is the accurate positional tracking of objects within the AR user's environment that is relevant in real-time applications. Multiple co-located frame based cameras are typical solutions proposed for object tracking in AR. The advent of commercial neuromorphic vision sensors (NVS) in recent years, introduce a novel, event-based approach to computer vision. Inspired by biological vision, these systems reduce the amount of data transferred and enhance power efficiency when tracking in real-time. This work leverages the strengths of AR and NVS technologies to develop an integrated system for tracking physical objects and providing users with required information to interact with these objects in a virtual environment. The computational pipeline from training on a deep neural network, coordinate transformations from image, to physical and virtual coordinate systems and testing the utility of the system for an AR system user is presented. The NVS system shows significant improvement over frame camera based systems in responding to varied lighting conditions.