Synthetic Hate Speech Data Underrepresents Minority Groups

The use of synthetic data is becoming increasingly widespread in a variety of applications. For hate speech detection, synthetic data offers the chance to overcome limitations of existing models and datasets, such as data scarcity, negative psychological impact on annotators, and privacy issues. However, previous work on the topic has only focused on evaluating synthetic datasets extrinsically, by measuring the performance of models trained on synthetic texts. In this work, in addition to measuring the performance of models trained on synthetic data in cross-distribution setups, we carry out a qualitative analysis, aiming at assessing how closely synthetic data can mimic authentic user-generated and human-annotated data. We focus in particular on possible issues related to (under)representation of specific minority groups as targets of hate in synthetic data. To do this, we augment an existing English hate speech dataset by paraphrasing the original gold data using three large language models, and then evaluate the synthetic data both quantitatively, including testing out-of-distribution model generalization, and qualitatively, examining the realism of synthetic texts, their hate speech content, and whether they represent the same identity groups as the original data. We pair this analysis with a detailed breakdown, for each demographic characteristic, of how the synthetic data creation process affects the representation of each specific type of identity, which is the main contribution of our work. We find that, while the performance of models trained on synthetic data is promising, these texts are easily identified as written by an LLM by humans, hate speech labels are not necessarily preserved, and the paraphrasing process tends to erase representation of minority groups and intersectional identities, affecting certain demographics more than others.