MultiSEAO-Mix: A Multimodal Multitask Framework for Sentiment, Emotion, Support, and Offensive Analysis in Code-Mixed Setting

Social media platforms have become an open door for users to share their views, resulting in a growing trend of offensive content being shared on social media. Detecting and addressing offensive content is crucial due to its significant impact on society. Although there has been extensive research on the detection of offensive content in the English language, there is a notable gap in detecting offensive content in multimodal settings involving code-mixed languages. In this article, we propose a large scale multimodal code-mixed dataset for Hinglish (Hindi+English) <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">MultiSEAO-Mix</i> focusing on women and children. The <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">MultiSEAO-Mix</i> is annotated with offensiveness, sentiment, emotion, and their respective intensities. Additionally, it is also annotated with author support. A multimodal, multitask framework is proposed that considers offensive detection, intensity prediction, and author support as the primary tasks and improves their performance using sentiment, emotion, and corresponding intensities as the auxiliary tasks. Further, we propose a fusion technique that captures the enhanced multimodal representation to improve the performance of our model. Experimental results demonstrate that the proposed multitask framework improves the model performance by more than 4.5 points compared to multitask system without sentiment and emotion as the auxiliary tasks.