When Do Parameter Budgets Misprice CPU Inference? A Matched-Budget Test of Byte-Traffic Pruning Allocation
Pruning budgets for language models are almost always stated in parameter count, yet CPU decode cost is plausibly driven by the bytes moved per token: weight bytes streamed each step plus KV-cache bytes read at the occupied context. We test, under a pre-registered protocol, whether allocating a 25% structured prune of Llama-3.2-3B by an analytical per-token byte-traffic model beats allocating it by parameter count, at fixed CPU thread caps (2 and 4 threads, both pinned to 2 cores). Seven allocations search one shared unit space (layers, FFN channels, KV groups) and differ only in the cost model used to choose the cut. The result is a bounded negative. In a held-out (leave-one-variant-out) prediction contest, the traffic model does not beat the parameter model: at 2 threads AIC (parameters - traffic) is +2.21 (the traffic model needed >10 to win) and at 4 threads -19.11 (parameters win outright). The predeclared byte allocation is not reliably faster than the parameter allocation to the registered standard: paired time ratios are 0.917 [0.824, 1.020] and 0.986 [0.934, 1.041] at 2 threads (8192 and 16384 context), and 0.818 [0.795, 0.842] and 0.817 [0.609, 1.096] at 4 threads, against a required upper confidence bound below 0.90, and the traffic model's own predicted ratios (0.771, 0.673) are systematically optimistic. Within this matched-budget CPU pruning family, an analytical weight-plus-KV traffic model therefore did not improve held-out prediction or registered selection over parameter count. The negative is about the analytical model, not about traffic-aware pruning as such: pruning by measured latency remains untouched by our result. Two scope limits bind the result: we measure decode speed only (pruned-model quality is not evaluated), and all timing is on a shared 2-core virtual machine whose tenant-load variance — including a documented load-era effect on the earliest cells — is a formal limitation. Descriptively, the byte-greedy pick's mean selection regret is low (3.8% and 1.4% at the two caps, vs 24.4% and 21.3% for the parameter pick) — a good pick, that the model itself does not explain. A post-hoc sensitivity excluding the saturated 16384 context reverses the 4-thread contest, localising the failure at the longest context; the frozen verdict is unchanged. We report the mechanisms consistent with the failure, so the boundary of the traffic model's usefulness is itself the contribution.
Authors
- Prabhu Rangaraju (ORCID: https://orcid.org/0009-0009-1861-1401)
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-10-26
- DOI
- https://doi.org/10.5281/zenodo.23670306
- Primary Topic
- Advanced Neural Network Applications
- Type
- preprint