Forecasting blink behavior during VR experiences via a multi-scale multi-modal fusion approach

Cybersickness induced by virtual reality (VR) applications remains one of the main obstacles to the widespread adoption of this technology. Despite extensive research on reducing cybersickness, there is still a lack of non-invasive methods to predict the severity of users’ cybersickness in advance. Considering the advancements in eye-tracking technology within VR head-mounted displays and previous studies on the correlation between blinking behavior and cybersickness, this study aims to propose a method for predicting users’ future blink behavior, thereby providing a foundation for subsequent non-invasive cybersickness prediction by leveraging the correlation between blinking and cybersickness. Based on an encoder–decoder architecture, we developed a multi-scale fusion deep neural network model using datasets collected from 23 participants performing virtual driving and free exploration tasks. The proposed model comprehensively considers users’ head and eye movement time-series data as well as multiple external features, and achieves future blink behavior prediction from the perspectives of both blink frequency and blink completeness. Extensive experiments and statistical analyses demonstrate that our model achieves high-precision, fine-grained blink behavior prediction over multiple future time intervals, providing a theoretical basis for future work on cybersickness prediction using blink behavior.