MTSA: Multi-turn Safety Alignment for LLMs through Multi-round Red-teaming

The proliferation of jailbreak attacks against large language models (LLMs) highlights the need for robust security measures.However, in multi-round dialogues, malicious intentions may be hidden in interactions, leading LLMs to be more prone to produce harmful responses.In this paper, we propose the Multi-Turn Safety Alignment (MTSA) framework, to address the challenge of securing LLMs in multi-round interactions.It consists of two stages: In the thought-guided attack learning stage, the redteam model learns about thought-guided multiround jailbreak attacks to generate adversarial prompts.In the adversarial iterative optimization stage, the red-team model and the target model continuously improve their respective capabilities in interaction.Furthermore, we introduce a multi-turn reinforcement learning algorithm based on future rewards to enhance the robustness of safety alignment.Experimental results show that the red-team model exhibits state-of-the-art attack capabilities, while the target model significantly improves its performance on safety benchmarks.Code is available at https: