Diversity Seeking Techniques for Red-Teaming Large Language Models

In this paper, we present new techniques for increasing the diversity of red-teaming prompts generated by automated machine learning-based methods, thereby enabling the discovery of more vulnerabilities in large language models. Using reinforcement learning to train models to output effective prompts for this task results in the models converging deterministically to a single output. Our first technique, which we term Defender, acts by blocking the reward signal for prompts that have already been discovered, thus making what was a stationary problem into a non-stationary problem that compels the reward maximizing algorithm to continually seek new prompts. Our second technique, Teamplay, trains two prompt generation models in tandem and adds the KL divergence between them to the reward in order to make them search in disparate regions of the space of prompts. Our techniques are shown experimentally to increase the effectiveness and diversity of prompts generated by existing reinforcement learning baselines.