Are Existing Large Language Models Robust Against Jailbreak Attacks?

The safety and robustness of Large Language Models (LLMs) are major challenges in developing generative AI applications. One key issue is the vulnerability to prompt jailbreak attacks, which pose a significant threat to building secure and resilient LLM-based applications. In this work, we present a framework for understanding and evaluating the behaviors of popular LLMs by categorizing their responses into five distinct exposure levels. Additionally, we introduce a novel language attack that circumvents LLMs’ defenses by translating jailbreak prompts into languages such as Arabic, Chinese, and Greek. Despite ongoing efforts to enhance LLMs’ safety, we find that nearly all popular LLMs can be jailbroken. Our findings offer detailed insights into LLMs’ behavior, improve diagnostic capabilities, and support targeted safety improvements.