Caste-Based Hate Speech Detection in Low-resource Hindi Language

In the digital era, social media discourse significantly shapes societal attitudes, frequently perpetuating entrenched hierarchies such as caste in India. Despite legal measures against caste-based discrimination, caste divisions continue to thrive online, fostering exclusion and hostility. To address this critical issue, we introduce the first curated Hindi-language dataset focused on caste related hate speech. Our annotation strategy employs a hybrid human-in-the-loop approach: expert annotators first labeled a subset of the data, which then served as a seed for large language model (LLM)-driven annotation of the remaining corpus. This methodology ensures both accuracy and scalability while capturing essential contextual and cultural nuances. We further benchmark a range of state-of-the-art hate speech detection models on our dataset, providing rigorous evaluation and insights into their performance on this challenging domain. Notably, we find that MuRIL, a multilingual language model tailored for Indian languages, achieves the best results on our test set, which consists of 761 non-hate speech and 712 hate speech comments. MuRIL attains an F1-score of 0.82 for hate speech, 0.84 for non-hate speech, and an overall accuracy of 0.83. These findings underscore the necessity of culturally and linguistically informed approaches for effective hate speech detection in low-resource settings. Our dataset, methodology and model insights offer a valuable foundation for further research in this domain, paving the way for the development of more nuanced and context-aware moderation tools. ContentWarning: This article contains illustrative examples of hateful content.