CLAIRE: a CNN–LSTM framework with adaptive and iterative refinement for multimodal emotion recognition

Recent breakthroughs in Deep Learning and bio signal processing have encouraged researchers to model human–computer interaction with greater accuracy. Emotion recognition is critical in facilitating the understanding and reaction of machines to the affective states of human beings. Many emotion recognition approaches have been suggested to identify emotions based on unimodal features. However, the use of unimodal features alone has resulted in more misclassification errors due to the presence of overlapping patterns among similar emotions. Although determining emotions in multimodal signal provides a higher recognition rate, it remains a challenging task. To address this challenge, this study proposes CLAIRE (CNN–LSTM with Adaptive and Iterative Refinement), a representation-level multimodal emotion recognition framework that independently learns modality-specific features from speech, EEG, and ECG using dedicated CNN–LSTM branches. Instead of performing direct sample-level fusion, the learned latent representations are projected into a shared embedding space and integrated through an adaptive fusion strategy for robust emotion classification. Furthermore, an Adaptive Grid Search Refinement (AGSR) algorithm is introduced to optimize the network hyperparameters efficiently while reducing computational overhead. Experimental evaluation demonstrates that the proposed framework achieves an overall emotion recognition accuracy of 97.2%, outperforming the considered baseline methods while exhibiting improved robustness and computational efficiency. These results demonstrate the effectiveness of representation-level fusion for heterogeneous emotion datasets and highlight the potential of the proposed framework for affective computing applications.