MultiFed-Emote: Privacy-Preserving Multimodal Emotion Recognition via Federated Fusion of Physiological, Speech, and Facial Signals

As human-computer interaction and affective computing advance, accurate emotion detection across diverse modalities remains critical for applications in healthcare, education, and immersive technologies. This work introduces MultiFed-Emote, a privacy-preserving system for multimodal emotion recognition that integrates speech (CREMA-D), facial expressions (FER-2013), and physiological signals (PhyMER) within a federated learning framework. To address challenges of data privacy and modality-specific limitations, we preprocess each dataset using audio standardization, image normalization with augmentation, and EEG/EDA filtering, and extract robust features including MFCCs, CNN embeddings, and Power Spectral Density values. The fused multimodal model uses an attention-driven hybrid approach to dynamically weight modality contributions. Experiments yield accuracies of approximately 67 percent for speech, 62 percent for facial, and 65 to 80 percent for physiological data in centralized settings, with federated fusion targeting 73 to 78 percent. Benchmarking against state-of-the-art models demonstrates competitive performance while preserving privacy, supporting scalable applications in personalized assistants and mental health monitoring.