PIGuard: Prompt Injection Guardrail via Mitigating Overdefense for Free

Prompt injection attacks pose a critical threat to large language models (LLMs), enabling goal hijacking and data leakage.Prompt guard models, though effective in defense, suffer from over-defense-falsely flagging benign inputs as malicious due to trigger word bias.To address this issue, we introduce NotInject, an evaluation dataset that systematically measures over-defense across various prompt guard models.NotInject contains 339 benign samples enriched with trigger words common in prompt injection attacks, enabling fine-grained evaluation.Our results show that state-of-the-art models suffer from over-defense issues, with accuracy dropping close to random guessing levels (60%).To mitigate this, we propose PIGuard, a novel prompt guard model that incorporates a new training strategy, Mitigating Over-defense for Free (MOF), which significantly reduces the bias on trigger words.PIGuard demonstrates state-of-the-art performance on diverse benchmarks including Not-Inject, surpassing the existing best model by 30.4%, offering a robust and open-source solution for detecting prompt injection attacks.