GM-Sparse-LM: Activation-Sparse FFN Execution for Memory-Bandwidth-Bound CPU Inference
Decoding a dense language model on a commodity PC with single-channel memory isbounded by RAM bandwidth, not compute. On an 8 GB single-channel DDR4-3200 laptop(about 11 GB/s effective decode bandwidth), a dense 4B Q4 model is capped at roughly4 tokens per second. This work measures the constraint on real hardware, studies FFNactivation sparsity on Qwen3-0.6B across all 28 layers, and proposes GM-Sparse-LM:an importance-sorted dual-precision FFN layout with masked cold-block execution anda tiered training pipeline (SiLU-to-ReLU conversion, ProSparse-style regularizedfine-tuning, calibration) projected to reach roughly 2.4x speedup at 90 percentsparsity. Measured contributions include a validated byte-level conversion path forGGUF, converted-model benchmarks (the expert-form model reads 25 percent of FFNbytes and gains 1.21x in the bandwidth-bound regime), a measured training-freequality cliff at roughly 37 percent sparsity that matches the sparsity study'soperating point, an implementation finding that explains why prior activation-sparsity systems all ship custom kernels, and eight quantified negative results.Trained-model quality evaluation is in progress.
Authors
- Sahil Arun (ORCID: https://orcid.org/0009-0007-3121-9370)
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-11-21
- DOI
- https://doi.org/10.5281/zenodo.24423661
- Primary Topic
- Advanced Neural Network Applications
- Type
- preprint