Addressing Past Tense Temporal Attacks in Large Language Models through Multiple Methods

Several previous studies highlight the problems that several state-of-art large language models are vulnerable under past tense attack (e.g.,” How to make a Molotov cocktail?” to” How did people make a Molotov cocktail?”). In this paper, we try several methods to address this problem on GPT-3.5-Turbo. The single prompt-engineering method reached a 1% attack success rate and a 17% over-refusal rate, and llama-2-7b trained on the Latent Adversarial Training method reached a 0% asr. However, its over-refusal rate was 48%. As an alternative method, we tried to build an advisor with a filter based on LLM, which can perform internet searching to check the safety of the content and provide more information with more timely related resources. With more information, the model can give more detailed insight even if it gives a possible refusal with “Sorry.” Source code at https://github.com/t0rit0/2024-summer-project