MASKSER: A Robust Emotion Recognition Model Based on Voice Data and Noisy Transcripts

In recent years, emotion recognition has become an increasingly vital tool for enhancing customer service applications. Especially in telephonic interactions, detecting emotions accurately is crucial for improving human-computer interaction experiences. Despite significant advances in deep learning, current emotion recognition systems that integrate voice and text face challenges such as noise interference in transcripts and inadequate multimodal fusion, which hinder precise emotion detection. In this paper, we introduce MASKSER, a methodology that combines vocal signals and transcribed text in a robust manner. Our approach involves pretraining noisy transcripts with ChatGPT-4 using few-shot learning based on techniques such as masking and sentiment word replacement. This enhances emotion discernment significantly by leveraging the strengths of both modalities. To address the challenges posed by noisy data, we propose a mask-based noise generation model and use it to pretrain the transcript-based model, which helps mitigate inaccuracies. Additionally, we introduce a novel loss function that evaluates the Kullback-Leibler divergence between text and voice encoder distributions, ensuring balanced contributions from both modalities. Experiments are conducted in both English and Korean to validate the language independence and robustness of the proposed approach in different linguistic contexts. The results demonstrate substantial improvements in emotion recognition capabilities, achieving high performance metrics while reducing reliance on costly speech recognition resources.