Multimodal Data Collection on Holographic 3D Objects Using HoloLens 2 and a Unity 3D Application for Perception Studies

Multimodal and visual attention data have been explored increasingly in training multimodal AI, the medical field, and human studies. Head mounted displays are widely used to collect multimodal and visual attention data in extended reality settings. While many programs have implemented Unity applications on HoloLens 2 for sensor data streaming, there is a lack of programs that are up-to-date, easily customizable, and geared towards presenting and recording multimodal participant responses on holographic 3D objects. In this work, we introduce a new framework to capture time-synchronized multimodal perception data in mixed reality environments. We implemented a Unity 2022 LTS 3D mixed reality (MR) application deployed on Microsoft HoloLens 2 for eye tracking, keypad response, and verbal impression data collection on 3D objects, such as archeological artefacts. We also implemented a data processing pipeline using Python to generate heatmaps and emotion maps for analysis. Furthermore, we conducted experiments to verify accuracy, precision, and inter-sample intervals by using a combination of 2D and 3D random saccade tasks and a continuous gaze tracking task. The best performance was achieved with 90 Hz sampling with a median inter-sample interval of 11.11 ms. Mean 2D resting tasks accuracy (1.58) and precision (0.89). 3D tasks obtained mean accuracy (1.56) and precision (1.68), both were similar to Microsoft's specifications (1.5–3 visual degrees). The framework can be applied to collect multimodal data in 3D extended reality studies.