Cross Modal Emotion Detection: Leveraging Speech and Facial Expression Features

Emotion recognition plays a vital role in enhancing human-computer interaction by enabling machines to interpret human affective states. This project proposes a cross-modal emotion detection system that leverages both speech and facial expressions to accurately classify emotions. The speech modality utilizes the RAVDESS dataset, where Mel-Spectrogram and MFCC features are extracted and processed through a combination of pre-trained DenseNet and custom CNN models. For the visual modality, facial expressions are analyzed using a Convolutional Neural Network trained on the FER2013 dataset, with real-time emotion detection enabled via live webcam feed.By integrating predictions from both modalities, the system improves the reliability and accuracy of emotion recognition compared to unimodal approaches. The final application is deployed as a user-friendly web interface using Streamlit, allowing users to upload audio files and capture live facial expressions for emotion analysis. This cross-modal approach demonstrates the effectiveness of multimodal learning in understanding complex human emotions and sets the foundation for more empathetic AI systems. Keywords— Emotion Detection, Speech Recognition, Facial Expression, Cross-Modal Fusion, BiLSTM, Machine Learning