A Consideration on Understanding of a Learning Based on EEG and ECG Analysis Using Artificial Intelligence Models

Many studies utilize biological signals such as electroencephalogram (EEG) and electrocardiogram (ECG) to estimate human states. Several studies show that combining multiple signals improves the accuracy of estimating complex states compared to using a single signal. However, the effective processing and integration of heterogeneous signal types continues to pose significant challenges. In this paper, a method is proposed to combine EEG and ECG with various artificial intelligence (AI) models to estimate and analyze learning understanding. In the proposed method, EEG and ECG features are extracted by various AI models, and these features are combined to classify the presence or absence of understanding. Convolutional neural networks (CNN) and EEGNet are employed for EEG feature extraction, and LSTM, GRU, and CNN are employed for ECG feature extraction. The combination patterns are all combinations of each feature extraction AI model. In experiments, participants answer “I understand” or “I do not understand” to an IQ test, while EEG and ECG are recorded. The proposed method is validated and analyzed with EEG and ECG obtained from experiments. In experimental results, the combined model performed best in precision <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$(\mathbf{75.7}\% \pm \mathbf{0.9}\%)$</tex> and sensitivity <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$(\mathbf{83.0}\% \pm \mathbf{2.0}\%)$</tex> when a CNN was employed to extract EEG and ECG features. In addition, the comparison of each pattern confirmed that there were differences in the tendency of misidentification data depending on the combined model. The results suggest the effectiveness of the CNN-based combination of EEG and ECG. Also, Comparison of each pattern revealed effective processing and issues for the combination of EEG and ECG, respectively.