CBRCL: A CLIP-BERT with Retrieval-Guided Contrastive Learning Multimodal Approach for Crisis-Driven Hate Speech Detection

The proliferation of multimodal text and image data has raised the challenge of detecting hate speech, a critical issue in mitigating the spread of hate and propaganda on social media. The challenge in detecting hateful posts arises when the combination of images and text, which individually lack hateful meaning, collectively convey hatefulness. To address this problem, we propose CBRCL, a novel approach leveraging CLIP and BERT for embedding generation and retrieval-guided contrastive learning to enhance hate speech detection. Our approach is evaluated on two datasets: CrisisHateMM, comprising text-embedded images associated with the Ukraine-Russia conflict, and LoveHate, focused on text-embedded images from the Israel-Palestine conflict. Experimental results demonstrate that CBRCL outperforms both unimodal and multimodal baselines, achieved an accuracy of 86.4% on CrisisHateMM and 76.3% on LoveHate datasets. By combining CLIP's strong visual-text alignment with BERT's deep contextual understanding, CBRCL effectively captures subtle multimodal hate speech. Additionally, retrieval-guided contrastive learning refines embedding spaces, ensuring a clearer distinction between hateful and non-hateful content.