Retrieval in the Voice Loop: Tool-Triggered RAG, Topic Preload and Caching Under a Latency Budget: Production Observations, a Design Pattern and an Evaluation Protocol

Retrieval-augmented generation (RAG) lets a voice agent answer from an organization's own documents, but every retrieval step sits between the caller finishing a question and hearing the answer. On a production voice platform, an early retrieval path took 8.2 s for one knowledge question: 1.8 s to embed the query, 3.3 s for vector search and 3.1 s for a separate synthesis model call. A redesigned path, which retrieves only when the conversation model calls a tool, returns raw passages instead of synthesized answers, caches results and optionally preloads common topics, answers warm queries in 0.2 to 0.5 s. This paper documents that redesign as a design pattern for retrieval under a voice latency budget, identifies two remaining defects found in production (synchronous retrieval calls that block a shared event loop, and retrieval calls with no timeout), and specifies a controlled evaluation. The evaluation compares five retrieval configurations, with and without caching, on a public document-grounded dialogue corpus, measuring added latency per turn together with answer quality, and measures event-loop blocking under 1 to 20 concurrent sessions. Hypotheses and the analysis plan are stated before data collection; results will be reported in a follow-up version of this preprint. Companion to: Rehman, A. (2026). Architecture Patterns for Low-Latency Real-Time Voice Agents. Zenodo. https://doi.org/10.5281/zenodo.23216507

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-12
DOI
https://doi.org/10.5281/zenodo.24173170
Primary Topic
Speech and dialogue systems
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Retrieval in the Voice Loop: Tool-Triggered RAG, Topic Preload and Caching Under a Latency Budget: Production Observations, a Design Pattern and an Evaluation Protocol

Altaf Rehman
Zenodo (CERN European Organization for Nuclear Research)
Speech and dialogue systems
preprint

Retrieval in the Voice Loop: Tool-Triggered RAG, Topic Preload and Caching Under a Latency Budget: Production Observations, a Design Pattern and an Evaluation Protocol

Altaf Rehman
preprint en

Abstract

Retrieval-augmented generation (RAG) lets a voice agent answer from an organization's own documents, but every retrieval step sits between the caller finishing a question and hearing the answer. On a production voice platform, an early retrieval path took 8.2 s for one knowledge question: 1.8 s to embed the query, 3.3 s for vector search and 3.1 s for a separate synthesis model call. A redesigned path, which retrieves only when the conversation model calls a tool, returns raw passages instead of synthesized answers, caches results and optionally preloads common topics, answers warm queries in 0.2 to 0.5 s. This paper documents that redesign as a design pattern for retrieval under a voice latency budget, identifies two remaining defects found in production (synchronous retrieval calls that block a shared event loop, and retrieval calls with no timeout), and specifies a controlled evaluation. The evaluation compares five retrieval configurations, with and without caching, on a public document-grounded dialogue corpus, measuring added latency per turn together with answer quality, and measures event-loop blocking under 1 to 20 concurrent sessions. Hypotheses and the analysis plan are stated before data collection; results will be reported in a follow-up version of this preprint. Companion to: Rehman, A. (2026). Architecture Patterns for Low-Latency Real-Time Voice Agents. Zenodo. https://doi.org/10.5281/zenodo.23216507

Zenodo (CERN European Organization for Nuclear Research)
Speech and dialogue systems
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon