Retrieval in the Voice Loop: Tool-Triggered RAG, Topic Preload and Caching Under a Latency Budget: Production Observations, a Design Pattern and an Evaluation Protocol
Retrieval-augmented generation (RAG) lets a voice agent answer from an organization's own documents, but every retrieval step sits between the caller finishing a question and hearing the answer. On a production voice platform, an early retrieval path took 8.2 s for one knowledge question: 1.8 s to embed the query, 3.3 s for vector search and 3.1 s for a separate synthesis model call. A redesigned path, which retrieves only when the conversation model calls a tool, returns raw passages instead of synthesized answers, caches results and optionally preloads common topics, answers warm queries in 0.2 to 0.5 s. This paper documents that redesign as a design pattern for retrieval under a voice latency budget, identifies two remaining defects found in production (synchronous retrieval calls that block a shared event loop, and retrieval calls with no timeout), and specifies a controlled evaluation. The evaluation compares five retrieval configurations, with and without caching, on a public document-grounded dialogue corpus, measuring added latency per turn together with answer quality, and measures event-loop blocking under 1 to 20 concurrent sessions. Hypotheses and the analysis plan are stated before data collection; results will be reported in a follow-up version of this preprint. Companion to: Rehman, A. (2026). Architecture Patterns for Low-Latency Real-Time Voice Agents. Zenodo. https://doi.org/10.5281/zenodo.23216507
Authors
- Altaf Rehman (ORCID: https://orcid.org/0009-0003-3414-4790)
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-11-12
- DOI
- https://doi.org/10.5281/zenodo.24173170
- Primary Topic
- Speech and dialogue systems
- Type
- preprint