Advanced transformer model with fine-grained correlation fusion for Multimodal Emotion analysis

In recent days, Emotion recognition has garnered a lot of attention because of its important applications in human-computer interaction. Human emotion is expressed through a variety of verbal and nonverbal languages, including text, audio, and visual. Current multimodal emotion analysis techniques typically perform poorly because they don't fully utilize the link between various modalities. An advanced transformer model with fine-grained correlation fusion (AT-FGCF) is presented in this research to identify emotion in text, audio, and visual modalities. To enhance the representation of the input audio and video, dense hierarchical feature maps are captured using the Dense Swin Transformer (DSwinT) model. Also, an Emotion- Aware Cognitive BERT (EAC-BERT) is introduced to enhance text representation. Lastly, a fine-grained correlation fusion is carried out to take advantage of correlations between fine-grained features from multimodal data. Our experimental findings show that the suggested method is successful in reaching radical outcomes across two benchmark datasets. That is the datasets IEMOCAP, and MELD all have average accuracy improvements of about 88.35%, and 84.80% respectively.