An Investigation on the Efficiency of Some Text Attack Algorithms

Machine learning models trained on human language, also known as Natural Language Processing (NLP) models, are susceptible to manipulation. These attacks, called NLP attacks, work by subtly altering the input text. This can involve swapping words, adding new ones, or removing existing ones. While these changes might seem insignificant to humans, they can fool the model into making incorrect predictions. The robustness of NLP models against adversarial attacks has become a critical area of research. This work delves into the effectiveness of three well-known text attack methods: PWWS, TextBugger, and TextFooler. The performance of these schemes is evaluated on two popular pre-trained transformers, DistilBERT and RoBERTa (both based on BERT). The experiments involve three text classification datasets: IMDB (movie review sentiment), AG News (topic classification), and SST2 (sentiment analysis). From each dataset, 100 examples are used to assess how effectively each attack method deceives the models and analyze the unique strengths and weaknesses of each approach. PWWS employs word saliency and probability weighting to generate adversarial examples, whereas TextBugger and TextFooler utilize semantic and syntactic perturbations to fool well-trained models. The results reveal the efficiencies of these attacks in different conditions. This analysis offers crucial insights into the susceptibilities of BERT-based models to prominent attack strategies, highlighting the need for enhanced defense mechanisms. By meticulously examining the attack vectors and their impacts, this research contributes to the broader understanding of adversarial attacks in NLP. It lays the groundwork for future advancements in model robustness.