Deceiving LLM through Compositional Instruction with Hidden Attacks

Recently, large language models (LLMs) have demonstrated promising applications in the autonomous driving (AD) domain, including language-based interactions and decision-making. Ensuring they safely handle harmful inputs is crucial before formal deployment. However, research reveals emerging hand-crafted jailbreak attacks, which pack harmful prompts into harmless instructions, can bypass LLMs’ security mechanisms and elicit harmful responses. To deeply understand such jailbreaks, this paper introduces a Compositional Instruction Attack (CIA) framework to generalize them, and develop two CIA jailbreaking methods to automatically generate tailored jailbreak prompts for each harmful prompt. Then, this paper builds the first CIA question-answering (CIAQA) dataset with 2.7K multiple-choice questions of 900 successful jailbreaks, for assessing LLMs’ ability to identify underlying harmful intents, harmfulness, and task priority in CIA jailbreaks. Combined with experimental analysis on CIAQA and other datasets, this paper concludes three possible reasons for the failure of LLM defenses against CIAs. Finally, we propose an intent-based defense paradigm (IBD), enabling LLMs to defend against CIA by leveraging its capability to identify intents. Experimental results show CIA can achieve attack success rates (ASR) of 95%+ and 85%+ in AD and common harmful scenarios for three well-known LLMs (GPT-4, GPT-3.5, and Llama2-70b-chat), and IBD reduces ASR by 74%+.