Niko OS NikoBench: Characterizing the End-to-End Performance, Resource, and Accuracy Tradeoffs of Local Desktop Voice Agents on Commodity Hardware

Cloud-hosted conversational AI assistants present data privacy vulnerabilities, high network latency dependencies, and recurring API token costs. While recent work has demonstrated on-device conversational pipelines and isolated edge-LLM inference components, systematic characterization of the combined performance, accuracy, resource, tool-routing, and network-isolation tradeoffs of complete local desktop voice-agent pipelines remains under-explored. We present Niko OS and NikoBench v9.2, an open-source desktop AI agent architecture and benchmark suite with runtime network-isolation monitoring designed for local execution on a consumer-grade 16 GB Apple Silicon MacBook Pro. Niko OS is the full desktop application (https://github.com/Ayjeren004/niko-os); NikoBench v9.2 is the standalone headless benchmark suite extracted from it for reproducible performance evaluation. We conduct an empirical study across 300 live pipeline executions over a 25-utterance multi-voice acoustic speech dataset on an Apple Silicon MacBook Pro (Apple M3, 16.0 GB Unified RAM), evaluating quantized open-weight Llama 3.2 models. Our empirical findings establish three contributions. First, among configurations achieving at least 80% RAG task success, Faster-Whisper-base with Nomic-Embed-Text and Llama-3.2:1b provided the best accuracy-efficiency tradeoff — reducing WER substantially relative to Whisper-tiny (15.99% vs. 24.00%) while maintaining a mean latency of approximately 2.0 s and a sub-1 GB peak memory footprint (705.6 MB). Second, our disaggregated failure taxonomy reveals that action name misalignment — not JSON syntax failure — is the dominant observed failure mode, accounting for 98% of all routing attempts in the whisper-small + llama3.2:3b configuration. Third, we propose and empirically evaluate a model-free inference-time hybrid semantic matcher that resolves action name misalignment, improving tool selection from 8.0% to 56.0% (+48.0 pp) without model retraining. A controlled ablation demonstrates that string-only fuzzy matching is ineffective (2% trigger rate, 0 pp improvement), while the full three-stage hybrid achieves a statistically significant improvement over fuzzy-only (Wilcoxon signed-rank, p=0.004, N=50 paired trials).