Evaluating the robustness of fake news detectors to adversarial attacks with real user comments

Abstract The widespread use of social media has led to an increase in false and misleading information presented as legitimate news, also known as fake news. This poses a significant threat to societal stability and has led to the development of fake news detectors that use machine learning to flag suspicious information. These automated systems analyze various aspects of news content, including text, images, and user interactions, to determine the likelihood of an article being fake or genuine. However, existing fake news detection models are vulnerable to attacks by malicious actors who can manipulate data to change predictions, potentially undermining the effectiveness of these crucial tools in combating misinformation. While considerable research has been conducted on attacks targeting news content itself, such as altering headlines or body text, research on attacks involving news comments is limited. User comments play a vital role in how news is perceived and shared on social media platforms, making them an attractive target for those seeking to manipulate fake news detection systems. Current attack methods focusing on comments are easily detectable, often relying on synthetically generated text that can be identified by secondary classifiers. This highlights the need for more sophisticated and harder-to-detect attack strategies that leverage authentic user-generated content. To address this gap, we propose two new attack strategies that instead use real, pre-existing comments from the same dataset as the news article to fool fake news detectors. Our first strategy involves selecting “topic-specific” comments that are thematically similar to the targeted news article while our second strategy consists of retrieving “generic” comments, which are highly dissimilar from their respective news articles. Extensive experimental results show that fake news detectors are less robust to our proposed attack strategies than existing methods using pre-existing human-written comments, as well as a malicious synthetic comment generator.