Kimi K3 Under Compute Constraints: Implementing MergeMoE to Take 14 Shards to 5GB on Consumer Hardware

This paper documents the end-to-end effort of reducing Kimi K3, a 2.8-trillion-parameter, 896-expert Mixture-of-Experts model, to a single serving-grade ~5 GB, 8-layer slice that runs on a 2019 Lenovo ThinkPad L390 laptop (Intel Core i5-8265U, 7.4 GB usable single-channel DDR4, no discrete GPU). The project proceeds in three acts: first, Unsloth's 594 GB, 14-shard 1-bit GGUF release is merged with a purpose-built expert-merging utility into a single dense 66.3 GB model (62.23 B parameters, two super-experts per layer); second, that model is pruned and quantized into an 8-layer serving slice through a path documented as a study in failure modes, including a silently corrupted quantization intermediate, a degenerate rebuild, a layer-pruning tool bug, and a metadata array mismatch; third, the working slice is measured exhaustively: ~2.5-2.6 tokens/s single-stream decode, ~5.3 tokens/s aggregate across six batched sequences, a ~458K tokens/day sustained ceiling, and a definitive single-channel-memory diagnosis from measured memcpy bandwidth. The central findings are that a frontier-scale MoE can be brought down to a running 5 GB artifact on decade-old consumer silicon, that the machine's memory subsystem (not the model or the kernels) is the binding constraint, and that the careful post-mortem of the corrupted-slice incident is the most transferable engineering output. Version 5 correction (Section 4.1): The MergeMoE accuracy claims were corrected. The paper previously stated that expert merging is near-lossless based on the MergeMoE paper, but MergeMoE only evaluates 2:1 compression ratios (e.g. 128 to 64 experts) on full-precision models. This project applies a 448:1 ratio (896 to 2 experts per layer) using a simpler weighted-average merge operator rather than MergeMoE's least-squares optimization. The actual quality retention is unknown and may be substantially lower than the ~77-78% upper bound previously cited. Line 448 was also corrected: the 36.4 GB variant referenced did not exist as a verified artifact; the full merged model is 66.3 GB all-Q8_0. Author: Haitham al-Waisy Contact: [email protected] | X/Twitter: https://x.com/HaithamalWaisy | YouTube: https://youtube.com/@haithamal-waisy