hearth-llm: Persistent KV-Cache Snapshots for Interactive Local LLM Inference

Local large language model (LLM) inference on consumer hardware suffersfrom a cold-context problem: every new session recomputes from scratchthe attention key-value (KV) cache for the entire context -- a codebase,a document, a long conversation -- even though the identical prefix wasprocessed moments or days earlier. On commodity accelerators thisprefill phase costs seconds to minutes and is discarded when theinference process exits. We present hearth-llm (hereafter hearth), alightweight system that persists KV-caches to local storage andrestores them across restarts.hearth operates as an OpenAI-compatible proxy in front of an unmodifiedllama.cpp server: it indexes conversation prefixes with a chain ofcontent hashes, restores the longest matching snapshot beforeforwarding each request, and saves the extended state in the backgroundafter each response completes. On an Apple M3 Max with a 7B-parametermodel, a cold session at a 30,294-token context takes 99.0 s, 97.9 s ofit prompt recomputation. A warm session with hearth takes 1.79 s: thesnapshot is read back from local storage in 0.46 s and only 16 prompttokens are computed instead of 30,294, a 55x end-to-end speedup. Themodel answers follow-up questions correctly from the restored state.Across six models from five families (1.5B-7B parameters), the fivedense-attention models show 20-32x speedups at a ~15k-token contextwith snapshot sizes matching each architecture's analytic KV footprintwithin 0.6%; a sliding-window model marks the limit of the approach:the engine cannot extend its restored prefixes past a divergencepoint, so warm sessions gain almost nothing.Snapshots cost 56.2 KB of disk per token and no resident memory betweensessions; the alternative, keeping the inference server alive, holds6.2 GB of memory the whole time. hearth is open source, ~1,100 linesof Python with no dependencies beyond the standard library, andrequires no changes to the engine or client.