A Novel Attention-Based Method for Integrating Facial and Speech Features in Emotion Recognition

Our research introduces a novel approach for recognizing multiple emotions by leveraging perspective-based methods. This technique integrates facial features and speech characteristics obtained from different encoders to pinpoint the most significant attributes. The distinct characteristics of speech and facial expressions, along with effective size management, prove to be highly beneficial. By concentrating on specific components, the system enhances overall accuracy. Images of facial features are captured from both low and high angles to ensure a comprehensive representation. These techniques are merged through a fusion mesh, producing several feature vectors that are subsequently utilized in the classification stage for sensitivity analysis. When tested with both data sets, this algorithm surpassed existing models, achieving a weighted accuracy (WA) of 90.3%.