Prompt Crossing: Evaluating Whether LLM Response Stem from Jailbreak or Normal Prompt

The evolution of Large Language Models (LLMs) has sparked growing concerns about jailbreak, crafted prompts that bypass safety guardrails and lead to the generation of harmful information. While recent research primarily focuses on identifying harmful content within LLM outputs, limited attention has been paid to detecting jailbreak intent directly from user prompts. This paper proposes a novel framework, called Prompt-Crossing, for analyzing user prompts and identifying those harboring potential jailbreak intent based on the response of LLM. Moreover, while existing methods determine jailbreak intent by looking at whether the response of LLM contains harmful information, our proposed framework can determine whether the user prompt, triggering the response of LLM, has jailbreak intent even if the response does not contain harmful information. In addition, our method performs filtering based on the likelihood calculated in a forward-step manner, deviates from traditional inference-based LLM usage, demonstrating a new application for LLMs. To demonstrate our proposed framework, we achieve state-of-the-art on the AdvBench benchmark using two LLMs, LLAMA and Falcon.