Intelligent ensemble classifiers for patient emotion recognition from multimodal data using weighted optimal score-based fusion of classifiers

Recent healthcare analytics increasingly employ deep learning for patient emotion recognition. However, conventional multimodal methods struggle to effectively capture complementary emotional information across text, video, and signal modalities. To address these limitations, a multimodal patient emotion recognition model integrating ensemble classifiers with weighted optimal score-based fusion is proposed. Initially, text, signal, and video data are acquired through standard resources. Text data are processed via Multihead Trans-Long Short-Term Memory (MHT-LSTM) to generate the first prediction score. Video data are processed using Spatio-Temporal Attention-based EfficientNetB7 (STA-ENetB7) to obtain the second prediction score, while signal data are processed using Spatio-Temporal Attention Autoencoder with LSTM (STA-AE-LSTM) to generate the third prediction score. These scores are integrated using Weighted Optimal-based Fusion of Ensemble Classifiers (WOS-FEC) to obtain the final emotion recognition result. The Refined Random Integer-based Modified Skill Optimization Algorithm (RRI-MSOA) determines optimal weights for individual classifier prediction scores. The proposed model achieves accuracy of 95.515% and lowest False Positive Rate (FPR) of 2.294% in 5th fold across all baseline algorithms and models. Overall, this study develops a multimodal ensemble framework integrating MHT-LSTM, STA-ENetB7, and STA-AE-LSTM, along with RRI-MSOA-based optimal weighting strategy. The WOS-FEC mechanism improves emotion recognition accuracy and reliability, enabling timely analysis in healthcare settings.

Authors

Institutions

Publication Details

Journal
Information Security Journal A Global Perspective
Published
2026-11-17
DOI
https://doi.org/10.1080/19393555.2026.2746748
Primary Topic
Emotion and Mood Recognition
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

Intelligent ensemble classifiers for patient emotion recognition from multimodal data using weighted optimal score-based fusion of classifiers

M.Chaitanya Bharathi, G. Rosline Nesa Kumari
1 links
Information Security Journal A Global Perspective
Emotion and Mood Recognition
article

Intelligent ensemble classifiers for patient emotion recognition from multimodal data using weighted optimal score-based fusion of classifiers

M.Chaitanya Bharathi, G. Rosline Nesa Kumari
article en

Abstract

Recent healthcare analytics increasingly employ deep learning for patient emotion recognition. However, conventional multimodal methods struggle to effectively capture complementary emotional information across text, video, and signal modalities. To address these limitations, a multimodal patient emotion recognition model integrating ensemble classifiers with weighted optimal score-based fusion is proposed. Initially, text, signal, and video data are acquired through standard resources. Text data are processed via Multihead Trans-Long Short-Term Memory (MHT-LSTM) to generate the first prediction score. Video data are processed using Spatio-Temporal Attention-based EfficientNetB7 (STA-ENetB7) to obtain the second prediction score, while signal data are processed using Spatio-Temporal Attention Autoencoder with LSTM (STA-AE-LSTM) to generate the third prediction score. These scores are integrated using Weighted Optimal-based Fusion of Ensemble Classifiers (WOS-FEC) to obtain the final emotion recognition result. The Refined Random Integer-based Modified Skill Optimization Algorithm (RRI-MSOA) determines optimal weights for individual classifier prediction scores. The proposed model achieves accuracy of 95.515% and lowest False Positive Rate (FPR) of 2.294% in 5th fold across all baseline algorithms and models. Overall, this study develops a multimodal ensemble framework integrating MHT-LSTM, STA-ENetB7, and STA-AE-LSTM, along with RRI-MSOA-based optimal weighting strategy. The WOS-FEC mechanism improves emotion recognition accuracy and reliability, enabling timely analysis in healthcare settings.

Information Security Journal A Global Perspective
Bharath University (IN)
Good health and well-being
Emotion and Mood Recognition
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon