A context-aware hybrid multimodal transformer framework for hate speech detection via contrastive annotation and graph-based semantic Expansion

Abstract Social media increasingly provides avenues for the distribution of nuanced, contextualized, and multimodal hate speech that is challenging to detect through traditional methods. Unimodal and superficial multimodal fusion methods have been observed to lack the ability to recognize implied hostility in sarcastic, satirical, figurative, meme, and visually symbolic messages. In addition, the noise in labelling data, limited generalizability of lexicons, and binary classification systems hinder the performance of current models in practical settings. To overcome the aforementioned issues, this paper proposes a context-aware hybrid multimodal model for hate speech recognition based on polarity-guided contrastive labelling, deep cross-modal transformer fusion, disagreement-aware labelling refinement, semantic graph augmentation, and hate intensity estimation. The designed framework leverages contextual language transformers for capturing complex semantic relationships in the text data while deep visual encoders capture emotion-specific and context-relevant features from images. Cross-attention based fusion facilitates alignment between text polarity and image emotion in the semantic embedding space, facilitating detection of implicitly embedded and image-assisted hateful sentiments. In order to overcome inconsistencies in annotation process and increase prediction accuracy, an ensemble-based disagreement correction method resolves the ambiguity in labelling by considering the impact of uncertainty and disagreement between multiple annotators. Concurrently, a semantic hate propagation network captures hate context information using graph-based lexicon relationships, enhancing hate detection in more sophisticated and less explicit hateful stories that existing keyword-based approaches miss out on. Unlike typical binary classification, the proposed approach implements a hierarchical hatred density score technique that classifies hatred content into ordinal scales, offering better interpretation and practical moderation results. In extensive evaluations using standard and mixed multimodal datasets, the proposed approach consistently outperforms the best-in-class method in terms of accuracy, precision, recall, and F1-score. The ablation study supports the positive role of all architectural components, while the computational analysis proves the viability for near real-time implementation. Finally, qualitative analysis highlights that the proposed approach generates explanations similar to human logic, especially for sophisticated scenarios of sarcasm, metaphorical hostility, and indirect hate speech.