Building a Robust Retrieval-Augmented Generation Chatbot for Immigration Knowledge Base using Google Cloud Vertex AI and Open-Source LLMs

This paper details the end-to-end development and implementation of a Retrieval-Augmented Generation (RAG) chatbot designed to provide accurate and grounded answers from a custom knowledge base on immigration policies and procedures. Leveraging Google Cloud’s Vertex AI for robust text embedding and vector search capabilities via Matching Engine, the system demonstrates an effective architecture for combating the inherent limitations of large language models (LLMs), such as factual inaccuracies and knowledge cut-off dates. The development journey highlights practical challenges encountered during LLM integration, including free-tier service access limitations with Google’s proprietary models and task-specific API constraints with hosted open-source solutions like Hugging Face’s Inference API. Ultimately, the project pivots to a strategy of local LLM deployment, providing valuable insights into the full lifecycle of a RAG system and the critical considerations for productionizing such applications within cloud environments, particularly concerning resource management and cost optimization.