Multimodal Infant Emotion Recognition Using Visual and Audio Features

Infant emotion recognition can help infants grow better. Due to the limited availability of multimodal datasets and the insufficient use of multimodal data in current infant emotion recognition research. First, we constructed an audiovisual infant emotion dataset with 1,338 samples, following the standards of Baby FACS and SAEVD-R, achieving an inter-rater reliability of 75%. Then, we propose a multimodal method that integrates visual and audio information to better capture the temporal dynamics of emotions, addressing the limitations of single-modal methods and improving recognition by learning key emotional segments. This method incorporates a forward masked temporal attention mechanism and a temporal attention enhancement module, allowing the model to capture dynamic relationships between current and past time steps while focusing on the most important video sub-segments for emotion recognition. Experimental results show that this method effectively extracts key features from the infant emotion dataset and the public adult audiovisual emotion dataset, RAVDESS, and significantly improves emotion recognition accuracy.