Niko OS NikoBench: Characterizing the End-to-End Performance, Resource, and Accuracy Tradeoffs of Local Desktop Voice Agents on Commodity Hardware

Cloud-hosted conversational AI assistants present data privacy vulnerabilities, high network latency dependencies, and recurring API token costs. While recent work has demonstrated on-device conversational pipelines and isolated edge-LLM inference components, systematic characterization of the combined performance, accuracy, resource, tool-routing, and network- isolation tradeos of complete local desktop voice-agent pipelines remains under-explored. We present Niko OS and NikoBench v9.2, an open-source desktop AI agent architecture and benchmark suite with runtime network-isolation monitoring designed for local execution on a consumer-grade 16 GB Apple Silicon MacBook Pro. Niko OS is the full desktop applica- tion (https://github.com/Ayjeren004/niko-os); NikoBench v9.2 is the standalone headless benchmark suite extracted from it for reproducible performance evaluation. We adopt inference-latency terminology and proling principles from ELANA (Chiang et al., 2025), combining them with standard speech-recognition (Radford et al., 2023; SYS- TRAN, 2024), retrieval (Lewis et al., 2020), and tool-routing (Schick et al., 2023) metrics: Word Error Rate (WER), Recall@k/MRR, vector retrieval latency, process-tree peak mem- ory, Speech-to-Synthesized-Audio latency (Tsza), and sampled runtime network-connection monitoring. We conduct an empirical study across 300 live pipeline executions over a 25- utterance multi-voice acoustic speech dataset on an Apple Silicon MacBook Pro (Apple M3, 16.0 GB Unied RAM), evaluating quantized open-weight Llama 3.2 models (Meta AI, 2024; Dubey et al., 2024). Our empirical ndings establish three contributions. First, among congurations achieving at least 80% RAG task success, Faster-Whisper-base with Nomic-Embed-Text and Llama- 3.2:1b had the lowest mean Tsza latency 1,914.0 ms, with 15.99% WER (vs. 24.00% for Whisper-tiny) and a sub-1 GB peak memory footprint (863.7 MB). Second, our disaggregated failure taxonomy reveals that action name misalignment not JSON syntax failure is the dominant observed failure mode across our benchmark congurations, accounting for 98% of all routing attempts in the whisper-small + llama3.2:3b conguration and establishing action disambiguation as the primary bottleneck in this evaluated local voice agent pipeline. Third, we propose and empirically evaluate a model-free inference-time hybrid semantic matcher that reduces action name misalignment, improving tool selection from 8.0% to 60.0% (+52.0 pp) without model retraining. A controlled comparison demonstrates that string-only fuzzy matching is ineective (2% trigger rate, −4 pp tool selection), while the full three-stage hybrid incorporating speech transcript context via reused nomic-embed- text embeddings achieves a consistent directional improvement in agent success (Wilcoxon signed-rank on stimulus means, p=0.0092, N=25). The hybrid matcher exposes argument generation as the next dominant failure mode, dening the primary open problem for local voice agent reliability.