Automated Boredom Recognition Using Multimodal Physiological Signals

Increasing access to affordable physiological sensors and computing devices has bolstered the development of emotionaware systems that recognize human emotions. However, to date, there has been little research testing machine-driven approaches to recognizing boredom, particularly in learning contexts. To address that, this study tests the automatic recognition of boredom induced during a video lecture using a multimodal recognition system relying on physiological signals. Electroencephalogram (EEG), electrocardiogram (ECG), galvanic skin response (GSR), and eye gaze data were recorded from 84 healthy adults (mean age = 26.90 ± 5.29) as they watched both non-boring and boring educational videos to control and induce boredom. Signal features were extracted from the physiological data and validated using Wilcoxon signed-rank tests prior to evaluating their utility for boredom recognition with three separate machine-learning classification techniques, namely Extreme Gradient Boosting (XGB), Random Forest (RF), and Gradient Boosting (GB), with leave-one-out cross-validation. Both unimodal and multimodal recognition systems were assessed by evaluating model performance using each physiological signal separately or in combination (using decision fusion). Multimodal recognition was found to enhance model performance when compared to any single modality, with the highest average boredom recognition accuracy of 88.56% ± 0.82% recorded for EEG + eye gaze modal fusion with RF.