Random Perturbation Attack on LLMs for Code Generation

Large language models (LLMs) have shown impressive capabilities in coding tasks, including code understanding and generation. However, these models are also susceptible to input perturbations, such as case changes, whitespace or typo modifications, which can affect their performance. This study investigates the impact of different types of perturbations on code and natural language on the performance of LLM code generation tasks. In addition to evaluating individual perturbations, the research examines combined perturbation attacks, where multiple perturbations from different categories are applied together. While combined attacks showed only marginal overall improvement over individual ones, they demonstrated a synergistic effect in specific scenarios, exploiting complementary vulnerabilities in the models.