Content Moderations without Labels: Unsupervised Classification of Hateful Texts

In today’s digital age, social media platforms serve as forums for diverse discussions. However, they also facilitate the spread of hate speech due to online anonymity and the lack of accountability. Islamophobia, a prevalent issue in this context, can have real-world consequences for Muslim communities. This study aims to assess how effectively we can classify social media text as Islamic and Islamophobic. Leveraging advanced natural language processing techniques, we explore the underlying topics in these discussions. Using Latent Dirichlet Allocation (LDA) and BERTopic as our topic models, we extract robust features from these posts to guide the classification process. A key aspect of our research involves the use of unsupervised text classification methods, enabling us to utilize vast amounts of unannotated social media data. For unsupervised text classification, we used Doc-SCAN and Lbl2Vec. We trained and evaluated these techniques on the two tasks of Islamic and Islamophobic Classification and on the two datasets of Twitter and Reddit posts. The combination of LDA as the topic model and Lbl2Vec as the classifier emerged as the most effective technique, consistently achieving the highest scores across all tasks and datasets. It yielded weighted average F1 scores of 89% on Reddit and 87% on Twitter for the Islamic classification task, and 82% on Twitter for the Islamophobic classification task.