Kringgående av stora språkmodellers säkerhet : en jämförande studie

Large Language Models (LLMs) are commonly finetuned with safety mechanisms to prevent the generation of harmful content. However, various jailbreak methods exist that aim to bypass these safeguards, often by altering prompts, model activations, or sampling methods. Existing safety evaluations focus mainly on Attack Success Rate, measured by refusal rates, and typically ignore both the quality of harmful responses and the broader impact on model capabilities and computational cost. This work proposes a comprehensive framework1 for evaluating jailbreak methods that consider harmfulness quality, effects on general model capabilities, and computational costs. A key methodological contribution is the use of an LLM-based classifier to assess response quality and harmfulness, revealing significant limitations in traditional refusal and substring-based evaluation methods. The evaluation demonstrates that current safety alignment mechanisms are more robust than previously characterized. Contrary to “catastrophic” claims in prior work, sampling-based exploits like Output-Aware Temperature Sampling show minimal effectiveness when properly evaluated, with much of their apparent danger stemming from inaccurate substring-based classifiers. While model manipulation methods such as Single Refusal Direction Ablation achieve higher safety bypass, they require substantial computational resources and often compromise general model capabilities. Importantly, refusal bypass rates consistently overestimate actual harmful damage, as simple refusal evasion does not necessarily translate to practical safety compromise. Our findings challenge prevailing narratives about jailbreak effectiveness and we emphasize the critical importance of comprehensive evaluation methodologies for accurate safety assessment.