Harnessing Response Consistency for Superior LLM Performance: The Promise and Peril of Answer-Augmented Prompting

This paper introduces Answer-Augmented Prompting (AAP), an innovative approach that leverages the Response Consistency of History of Dialogue (HoD) phenomenon in Large Language Models (LLMs). AAP not only achieves significantly superior performance enhancements compared to traditional augmentation methods but also exhibits a stronger potential for “jailbreaking”, allowing models to produce unsafe or misleading responses. By strategically modifying the HoD, AAP influences LLM performance in a dual manner: it promotes accuracy while amplifying risks associated with bypassing built-in safeguards. Our experiments demonstrate that AAP outperforms standard methods in both effectiveness and the ability to elicit harmful content. To address these risks, we propose comprehensive mitigation strategies for both LLM service providers and end-users. This research offers valuable insights into the implications of Response Consistency in LLMs, underscoring the promise and peril of this powerful capability.