On the Effectiveness of Text and Image Embeddings in Multimodal Hate Speech Detection

Social media content is increasingly subject to hate speech towards specific demographic groups. In this context, machine learning approaches appear relevant to detect malicious contents and support moderators in mitigation initiatives. However, many of the existing approaches are exclusively focused on the analysis of textual contents. On the other hand, studies that address multiple data modalities often rely on a single feature representation and a fully supervised learning setting. In this paper, we tackle multimodal hate speech detection resorting to different learning settings (one-class learning and binary classification). We also investigate the effectiveness of multiple deep learning model backbones and language models to extract embedding feature representations for text and image modalities. Our experiments with a real-world hate speech dataset show that there is a significant performance gap between one-class learning and binary classification, and that the choice of embedding representations for image and text modalities can impact the detection performance for different predictive models.