Designing and Implementing LLM Guardrails Components in Production Environments

With advancements in generative Artificial Intelligence (AI), there has been an increasing need for tools that rely on Large Language Model (LLM)s. As these models may produce undesired answers, there is a need to prevent such events, especially in enterprise environments. Even if models are trained on safe data, user inputs and even model behavior can be unpredictable, leading to problems such as leakage of confidential data that could result in revenue loss. In this paper, we describe our experiences on developing tools for “guardrailing” LLMs. We describe how we started with a quick monolith implementation, and later transitioned to a microservices architecture. As results, we share our lessons learned throughout the process, and how the re-architecture to microservices led to runtime performance gains, easier maintenance and extensibility, and also allowed us to open source the main component of the solution, so anyone can contribute to and use it.