Benchmarking LLM Robustness Against Prompt-Based Adversarial Attacks

As Large Language Models (LLMs) become increasingly integrated into real-world applications and are made more accessible to the general public, significant concerns regarding their security and the safety of the content they output arise. Recent research demonstrates that these models are vulnerable to adversarial techniques, which can manipulate them into generating harmful or biased outputs. These risks highlight the need for robust evaluation methods to assess the resilience of LLMs against these threats. While some efforts have been done to benchmark LLMs' robustness against adversarial prompt-based attacks, these approaches are often highly context-specific and lack the comprehensive components required for a robust and systematic evaluation. In this work we present a benchmarking framework for systematically evaluating the adversarial robustness of LLMs' built-in security and safety measures against text-based prompt attacks during inference time. The proposed framework is composed of the key components that make up a reliable benchmark, including scenarios, workloads, metrics and attack loads and is designed to ensure its representativeness, usefulness and adaptability to different contexts.