Conditional Transformer-Based U-Net Architecture for Speech Emotion Recognition

In this paper, we introduce a conditional transformer-based U-net architecture for speech emotion recognition (SER). The proposed architecture consists of convolutional transformer (CTR)-based encoder, feature decoder and emotion decoder. The CTR-based U-net encoder is designed to extract compressed bottleneck features for recognition. The emotion is then predicted by the CTR-based emotion decoder, where we use additional trainable emotion query to better capture the characteristics of different emotional states. The CTR-based feature decoder reconstructs the given input features from the bottleneck features. This auxiliary decoder allows us to use additional information while training the model, which further improves the recognition performance. Specifically, the proposed feature decoder is performed through sophisticated embedding of the predicted emotional states and the features of the encoder layers. In the proposed framework, we consider the channel attention for each time frame in the CTRs to better enable a real-time processing of emotion recognition. Experimental results showed that the proposed emotion classification method provided better recognition performance than the selected benchmark algorithms.