Advancing text adversarial example generation using large language models

• Novel method for generating text adversarial examples for machine learning models. • Adversarial examples achieved by utilizing recent large language models. • Facilitates systematic vulnerability analysis and improvement of model robustness. • Promising results in attacking recent open-weight large language models. Recent advances in Natural Language Processing (NLP) are highly based on black-box score-based models that provide only final predictions along with their score. This opacity impedes the comprehension of their internal decision-making processes, complicating the identification of potential weaknesses. A powerful strategy for analyzing model vulnerabilities is the generation of text adversarial examples. These attacks introduce subtle text perturbations that cause victim models to make incorrect predictions while preserving the original semantic meaning. This paper presents a novel method for generating text adversarial examples through Large Language Models (LLMs). The proposed method uses the outstanding text generation capabilities of LLMs to modify the original input text at multiple granularities: character, word, and sentence level. First, sentence-level perturbations are introduced by generating paraphrases with an LLM instruction prompt. Next, further character- and word-level perturbations are introduced to words that most affect predictions using another set of LLM instruction prompts. In particular, vulnerable words are perturbed by replacing them with their synonyms or misspelled variants, or by inserting additional neutral words adjacent to them. Experiments were conducted to assess the proposal’s viability on two sentiment classification tasks: sentence-level reviews and full-length reviews. The proposal demonstrates an advantage over many well-known approaches based on LLMs. It preserves the original semantics to a similar extent, while increasing the deception of victim models by 29 − 85 % over the best-analyzed state-of-the-art methods.