Activation-based efficient routing for retrieval-augmented generation in LLMs

Retrieval-augmented generation (RAG) enhances large language models (LLMs) by integrating relevant information from external sources, thereby improving the accuracy and reliability of outputs. Meanwhile, routing-based frameworks that coordinate multiple LLMs have extended the capability boundaries of large language models. However, when applied to domains that contain sensitive data, such as healthcare, finance, or proprietary expert models, RAG poses substantial privacy risks. We propose Vector-DiM, a privacy-first routing framework with training-free centroid-based updates that dispatches user queries to the most relevant expert models based on semantic embeddings using the Difference-in-Means (DiM) method, while transmitting only low-dimensional vector representations to protect privacy. Specifically, our system first utilizes a pre-trained LLM to generate a low-dimensional semantic embedding of the input query and constructs domain-specific semantic centroids using the DiM method. The query embedding is then compared with these centroids in the embedding space to identify the most relevant expert domain, and only the selected expert model receives the original text for final response generation. Throughout this process, intermediate modules exchange only low-dimensional vector representations, thereby reducing routing-time exposure of plaintext query content and private expert-side information. Extensive experimental results on MMLU and MMLU-Pro demonstrate that Vector-DiM achieves robust routing performance and preserves strong interpretability, low calibration cost, and favorable privacy characteristics, while supporting domain updates through centroid recomputation rather than task-specific router retraining.