The perils of stealthy data poisoning attacks in misogynistic content moderation

• The vulnerability of text classifiers (BERT, DistilBERT, XLNet) to targeted data poisoning attacks is analysed in the context of online harm detection, with a focus on misogynistic comments detection on Reddit. • A novel and stealthy poisoning strategy is introduced. Such strategy leverages the TracIn concept of opponents , i.e. training samples that negatively influence the prediction of a specific test point which are repurposed as poisoned inputs during training. • The designed data poisoning attack is effective in flipping the labels of target points with small number of poisoned instances. Crucially, the attack does not degrade the overall model performance, thus making its detection non-trivial. • The proposed data poisoning strategy is showed to be transferable across different model architectures, highlighting its practical relevance in real-world settings in which an attacker has no access to the target model. Moderating harmful content, such as misogynistic language, is essential to ensure safety and well-being in online spaces. To this end, text classification models have been used to detect toxic content, especially in communities that are known to promote violence and radicalization in the real world, such as the incel movement. However, these models remain vulnerable to targeted data poisoning attacks. In this work, we present a realistic targeted poisoning strategy in which an adversary aims at misclassifying specific misogynistic comments in order to evade detection. While prior approaches craft poisoned samples with explicit trigger phrases, our method relies exclusively on existing training data. In particular, we repurpose the concept of opponents , training points that negatively influence the prediction of a target test point, to identify poisoned points to be added to the training set, either in their original form or in a paraphrased variant. The effectiveness of the attack is then measured on several aspects: success rate, number of poisoned samples required, and preservation of the overall model performance. Our results on two different datasets show that only a small fraction of malicious inputs are possibly sufficient to undermine classification of a target sample, while leaving the model performance on non-target points virtually unaffected, revealing the stealthy nature of the attack. Finally, we show that the attack can be transferred across different models, thus highlighting its practical relevance in real-world scenarios. Overall, our work raises awareness on the vulnerability of powerful machine learning models to data poisoning attacks, and will possibly encourage the development of efficient defense and mitigation techniques to strengthen the security of automated moderation systems.