Multimodal confusion state detection in Japanese text reading using Vision Transformers, facial video, and eye tracking

• We proposed Vision Transformer based multimodal simple method using data overwritten outperforms a late fusion model. • Using novel eye gaze features transformation process, and data augmentation method. • Proposed method can train with small dataset (around 400 samples). The COVID-19 pandemic has led to a significant increase in online classes, necessitating robust support system for students in virtual learning environments. One critical aspect of such support system is the detection of confusion during text-based learning, as students may struggle with comprehension. In this study, we propose a multimodal data fusion approach leveraging machine learning models to detect confusion states in Japanese text reading. The proposed method integrates facial video and eye-tracking data using a Vision Transformer-based video classification model. The novelty of our approach lies in the early-stage data fusion, performed prior to model input, where facial images are augmented with eye-tracking features. Experimental results demonstrate that this fusion strategy, achieved through facial image overwriting, enhances confusion state detection accuracy compared to conventional late fusion machine learning models. These findings highlight the potential of the proposed Vision Transformer-based method as a simple yet effective framework for multimodal data analysis using image-based overwriting.