Pipit: A Placement-First Inference Engine for Frontier Mixture-of-Experts Models on Consumer Hardware

Mixture-of-experts (MoE) language models with hundreds of billions of parameters activate only a few per cent of their weights for each token. That makes them runnable on machines with far less memory than the model, provided the right expert weights sit in the right memory tier at the right moment. Today's local engines each solve part of this. Colibri streams experts from SSD but relies on general-purpose kernels and has repeatedly patched memory-safety defects in its loaders; uzu reaches strong speculative-decoding speed on Apple Silicon but is not yet stable on other operating systems; Magnitude tunes kernels per device and connects to coding agents, though its headline speed-ups rest on comparisons with mismatched settings. This report presents the design of Pipit, a memory-safe Rust engine that treats decoding as a bandwidth problem across three tiers (GPU memory, RAM and NVMe). It combines a slot-slab expert store, a learned expert predictor, bandwidth-saturating CPU kernels for six instruction-set levels, union-aware speculative decoding, and validated loaders for GGUF, safetensors and a new streaming format, KXF. We pre-register ten falsifiable hypotheses against same-machine baselines, together with a protocol that scores every run against its physical roofline. The report contains no measured Pipit results: it fixes the claims in advance, so that forthcoming results can be judged against them.