Messaging Maneuvers: Generating and Evaluating Strategic Counternarratives with Large Language Models

As Large Language Models (LLMs) become increasingly embedded in content moderation and public communication, their potential to both generate and evaluate strategic counterspeech demands close study. Our work introduces a novel pipeline for producing contextualized counterspeech aligned with the BEND maneuver taxonomy, and evaluates the quality and effectiveness of generated countermessages by using an LLM-as-a-judge guided by Tree-of-Thought prompting. We compare automated LLM-as-a-judge evaluations against human surveys grounded in cognitive science. We analyze their alignment across multiple dimensions that help us assess the overall persuasiveness of the BEND aligned counterspeech. LLM-as-a-judge and human feedback have strong alignment in particular to messages human raters find dismissive or distortive. This indicates the LLM-as-a-judge is well aligned for detecting manipulation, incoherence, and threatening behaviors. There is also significant perceived persuasiveness alignment for explanatory messages, which suggests that these types of messages have the potential to change people's attitudes and behaviors.