Improving the Hate Speech Annotation Process with Large Language Models

Hate speech detection in online environments poses significant challenges due to the nuanced and evolving nature of hate speech and the complexities of real-time communication. This study evaluates whether zero-shot large language models (LLMs), including Claude, Gemini 2.0, ChatGPT-o1, and ChatGPT-4o, can support the annotation of gaming-specific hate speech. We compare individual models, a multi-model collective, and TF-IDF–Logistic Regression classifier against an operational human-majority reference on 1,178 Dota 2 chat messages. We also examine whether ChatGPT-generated messages improve the traditional classifier when used exclusively as training-time augmentation, with full-archive augmentation and SMOTE as comparators. Among the automated approaches, the four-model collective achieved the highest accuracy and macro-F1 but did not uniformly outperform the strongest individual LLMs. For the class-sensitive measures, synthetic-data composition mattered more than volume. A post-hoc qualitative analysis identified recurring disagreement patterns and model-specific difficulties. Targeted augmentation achieved higher accuracy than SMOTE. The findings show that class-aware LLM generation can provide a competitive alternative to feature-space oversampling for lightweight classifiers, while zero-shot LLMs are best positioned as complementary annotation tools under human oversight.