MemHateCaptioning: Enhancing Hate Speech Detection in Memes with Context-Aware Captioning and Chain-of-Thought

Hate speech has become increasingly prevalent on social media, with memes presenting a unique challenge due to their multimodal nature, combining text, images, and often subtle cultural cues that spread hate online. Several hate speech detection models have recently been proposed to identify hate in multimodal memes. However, these methods may suffer from issues like context ambiguity, subtle visual cues, and multimodal complexity. Generating accurate captions from memes while considering the context, images, symbols, and text can help capture the intended meaning and, consequently, improve the accuracy of hate speech detection. This study introduces MemHateCaptioning, a framework designed to generate clear, human-like explanations to contextualize why a meme is flagged as hateful. MemHateCaptioning leverages recent advancements in vision-language models (VLMs) and large language models (LLMs), integrating ClipCap for image captioning, BLIP for language-image pretraining, and T5 for explanation generation. The framework incorporates Chain-of-Thought (CoT) prompting to enhance interpretability, enabling the model to break down complex reasoning step by step, which helps in comprehending the subtle interplay between text and images in hateful content. MemHateCaptioning is evaluated on the HatReD dataset and demonstrates strong performance, achieving higher BLEU and ROUGE-L scores compared to existing models. It also effectively reduces issues such as hallucinations and context misinterpretation by providing detailed, context-aware explanations.