Multi-Modal, Accessible Mixed Reality Framework for Object Exploration and Accompanying Hardware Exploration

With our research, propose a multi-modal Mixed Reality (MR) system framework with glove-based finger and hand movement tracking and tactile feedback, allowing blind and visually impaired individuals to have robust interactions with virtual objects that are not physically present. Our proposed framework will support interactions not only through haptics but also through other modalities including a personalized voice assistant audio interface, and visual interfaces. The several interaction modalities in our proposed MR framework and glove-based system would allow BVI users to experience virtual objects of any 3D model in greater depth without being constrained by their limited vision. However, contrary to common belief, most BVI are not fully blind and even prefer vision over other senses when possible. This means that visuals are still important and need to be used, though the system will be specifically designed to not completely rely on them. Our system framework’s features and modalities will both be configurable and thus enable a more customized experience that can be adapted to the various dynamic needs of both regular users and BVI users. In addition to successful implementation of the system framework, we propose the following research directions: explorations of the application of Machine Learning (ML) and Artificial Intelligence (AI) on sensor data for hand movement tracking, 3D model data, and voice data within the context of our framework, exploration of different haptic device implementations, and the exploration of hardware design for smartphone-based Augmented Reality (AR) Head-Mounted Displays (HMDs).