Advancing multimodal emotion analysis: integrating machine learning and deep learning approaches

Multimodal emotion analysis, blending machine learning and deep learning, is transforming computer-based human emotion recognition. This review examines the complexity of human emotions, generally classified into six types: joy, sadness, anger, fear, surprise, and disgust. It covers both unimodal and multimodal techniques, emphasizing their utility in fields like medicine, driving safety, and law enforcement. Unimodal technologies analyze single data types (text, image, audio), while multimodal fusion algorithms integrate these for heightened accuracy. Traditional machine learning methods, such as SVM and k-NN, are foundational for feature extraction and classification. Deep learning, using CNN and RNN architectures, advances emotion recognition by extracting intricate data patterns. Combining traditional machine learning with deep learning enhances precision, with the CNN-LSVM approach notably boosting classification accuracy. The paper focuses on three primary multimodal techniques: feature-level, decision-level, and hybrid fusion. Feature-level fusion merges various modality features to enrich classifier inputs. Decision-level fusion consolidates outcomes from different modalities, ensuring stable classification. Hybrid fusion, merging the prior two, offers a holistic approach for accurate emotion analysis. These methods, however, face challenges like increased computational demands and complex model training.