NSKVCache: 208x KV Cache Compression; Making AI Inference Economically Viable at Any Scale.

AI Inference can be significantly cheaper. NSKVCache is a retrieval-based KV cache compression system for long-context LLM inference. Unlike SnapKV, H2O, and StreamingLLM, which permanently discard tokens based on attention proxies, NSKVCache pages cold tokens to an INT8 compressed store and retrieves them on demand. Retrieval uses a blended QUEST and content-cosine scoring scheme, with K vectors stored pre-RoPE to eliminate distance suppression on injection. Results: 208.7x compression on CPU (Qwen2.5-3B, 1M context: 35,156 MiB to 168.5 MiB) and 66x on GPU (Nemotron-Nano-8B, RTX A5000, 1M context: ~140 GB to 2.1 GB, 5.4 GB total VRAM including the model + 1M context). Prefill throughput reaches 1823 t/s at 1M context on GPU. Cross-window needle recall confirmed (MANGO7734, content score 0.939 vs 0.888 filler). This is inside NSRun, part of the Type 1 Civilization Toolkit. For more information, please visit: Our website - ofmagnitude.com Our Github - https://github.com/OrdersOfMagnitudeLLC/type1-toolkit NSRun is a separate repository at github.com/OrdersOfMagnitudeLLC/NSRun