REAPER: Reasoning based Retrieval Planning for Complex RAG Systems

Complex dialog systems often use retrieved evidence to facilitate factual responses. Such RAG (Retrieval Augmented Generation) systems retrieve from heterogeneous data stores that are architected as multiple indexes or APIs instead of a single monolithic source. For a given query, relevant evidence needs to be retrieved from one (or few) retrieval source. Complex queries can even require multi-step retrieval. For example, a conversational agent on a retail site answering customer questions about past orders need to retrieve the appropriate customer order first and then the evidence relevant to the customer's question in the context of the ordered product. Most RAG Agents handle such Chain-of-Thought (CoT) tasks by interleaving reasoning and retrieval steps. However, each reasoning step directly adds to the latency of the system. For large models this latency cost is significant -- in the order of multiple seconds. Multi-agent systems may classify the query to a single Agent associated with a retrieval source, which means that a (small) classification model dictates the performance of a large language model. To address this problem, we present REAPER (REAsoning-based PlannER), an LLM-based retrieval planner that we evaluate on a conversational shopping assistant, which shows significant gains in latency over Agent-based systems and scalability to new and unseen use cases when compared to classification-based planning.