Prevention of Prompt Injection Attacks Over Financial Applications Integrated with LLM

Prompt injection is a growing concern in financial applications integrated with Large Language Models. These attacks pose a critical risk to financial applications leading to data breaches, confidential data loss and financial losses by manipulating input prompts to cause unintended or harmful outputs. Existing defense strategies include utilizing filters f or input and output, and delimiters, however, these techniques have been proven inadequate. This paper builds upon existing mechanisms and proposes an enhanced security system that incorporates role-based access, jailbreak attack detection, validating inputs and securely passing prompts to LLM. This is done by developing a pre-processing layer at both ends-the client and LLMs-through the identification of critical as well as destructive keywords. The experimental results show that blocking of that malicious prompt ensures that only authenticated and authorized commands are processed. The measures, therefore, allow financial institutions to significantly reduce the probability of prompt injection attacks besides increasing data protection and confidence i n AI-based financial applications.