Predicting Locomotion Intention using Eye Movements and EEG with LSTM and Transformers

Predicting future locomotion based on intrinsic data serves many purposes, including optimizing the utilization of physical space in virtual reality environments and enhancing the control of electronic aids for patients with motor impairments. However, predicting human locomotion intentions proves challenging due to the inherent difficulty arising from the highly complex and nonlinear interactions among the relevant parameters. Deep neural networks offer a significant advantage over conventional approaches in addressing this challenge. We treat this task as a time series prediction problem and compare LSTM networks to transformer models. A distinctive aspect of our work is our approach’s emphasis on eye movements as a central feature, contributing to its novel predictive capabilities. Besides gaze data, we evaluate the addition of EEG as a data source for this prediction task to be used in brain-computer interfaces. To achieve this, we conducted two data collection experiments in custom virtual environments that feature different tasks utilizing joystick control. We present these novel datasets in conjunction with this work. The results demonstrate that gaze data proves to be a valuable tool for locomotion prediction in different contexts, even when there is not a strong and direct connection between gaze and future waypoints. Transformer models were able to achieve better performance than LSTM networks, and we conclude that successful prediction across diverse situations requires datasets containing a wide range of movement scenarios.