Who Needs STT or TTS? The Waveform Remembers: Audio Memory for Voice Agents

This paper proposes an audio memory architecture for a voice assistant whose persistent conversational history is managed outside the response model. The architecture retains source recordings, indexes them with audio embeddings, retrieves relevant episodes, and supplies original audio to a compatible answering model. It extends this foundation with storage of the assistant's own speech, anticipatory retrieval, background consolidation, contradiction review, and optional transfer of stable experience into a trainable local adapter. These extensions form a design proposal; the empirical contribution is a narrower end-to-end API proof of concept. On 9 November 2026, a prerecorded synthetic-audio experiment combined Gemini Embedding 2, a local SQLite store, Gemini audio-pair review, and xAI spoken responses with the Ara voice requested. All six fresh final cases retrieved the required recordings, produced the intended review decisions, and generated spoken outputs. A frozen lexical answer check passed five of six cases; a separate posthoc inspection of diagnostic transcripts found the expected answer content in all six. Mean time to first audio was 10.704 seconds, including a 7.205-second blocking review that did not influence the voice model's evidence. The experiment establishes feasibility of the recorded-audio retrieval and response path and supports the broader architecture proposed here.