RAGs to Riches: Cost-efficient Complex Query Orchestration

Large Language Models (LLMs) have become integral to modern business operations, especially for tasks involving reasoning over large datasets. One prominent application of LLMs is in chatbot systems, where customers provide natural language queries, often complex in nature, requiring decomposition to retrieve relevant information from various data sources. These queries may span structured databases, unstructured data, or public information from the internet, making efficient data retrieval and reasoning vital for real-time, accurate responses. In this paper, we propose two cost-efficient ''Query Orchestration'' approaches (Context Latent and Context Acute) to address these challenges. By leveraging graph-based retrieval-augmented generation (RAG) techniques for vector search, we optimize data retrieval while minimizing reliance on LLMs for reasoning to reduce costs. Our approach is validated through experiments on a banking use case, where we demonstrate its effectiveness in providing high-quality