When Do Parameter Budgets Misprice CPU Inference? A Matched-Budget Test of Byte-Traffic Pruning Allocation

Pruning budgets for language models are almost always stated in parameter count, yet CPU decode cost is plausibly driven by the bytes moved per token: weight bytes streamed each step plus KV-cache bytes read at the occupied context. We test, under a pre-registered protocol, whether allocating a 25% structured prune of Llama-3.2-3B by an analytical per-token byte-traffic model beats allocating it by parameter count, at fixed CPU thread caps (2 and 4 threads, both pinned to 2 cores). Seven allocations search one shared unit space (layers, FFN channels, KV groups) and differ only in the cost model used to choose the cut. The result is a bounded negative. In a held-out (leave-one-variant-out) prediction contest, the traffic model does not beat the parameter model: at 2 threads AIC (parameters - traffic) is +2.21 (the traffic model needed >10 to win) and at 4 threads -19.11 (parameters win outright). The predeclared byte allocation is not reliably faster than the parameter allocation to the registered standard: paired time ratios are 0.917 [0.824, 1.020] and 0.986 [0.934, 1.041] at 2 threads (8192 and 16384 context), and 0.818 [0.795, 0.842] and 0.817 [0.609, 1.096] at 4 threads, against a required upper confidence bound below 0.90, and the traffic model's own predicted ratios (0.771, 0.673) are systematically optimistic. Within this matched-budget CPU pruning family, an analytical weight-plus-KV traffic model therefore did not improve held-out prediction or registered selection over parameter count. The negative is about the analytical model, not about traffic-aware pruning as such: pruning by measured latency remains untouched by our result. Two scope limits bind the result: we measure decode speed only (pruned-model quality is not evaluated), and all timing is on a shared 2-core virtual machine whose tenant-load variance — including a documented load-era effect on the earliest cells — is a formal limitation. Descriptively, the byte-greedy pick's mean selection regret is low (3.8% and 1.4% at the two caps, vs 24.4% and 21.3% for the parameter pick) — a good pick, that the model itself does not explain. A post-hoc sensitivity excluding the saturated 16384 context reverses the 4-thread contest, localising the failure at the longest context; the frozen verdict is unchanged. We report the mechanisms consistent with the failure, so the boundary of the traffic model's usefulness is itself the contribution.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-10-26
DOI
https://doi.org/10.5281/zenodo.23670306
Primary Topic
Advanced Neural Network Applications
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
preprint

When Do Parameter Budgets Misprice CPU Inference? A Matched-Budget Test of Byte-Traffic Pruning Allocation

Prabhu Rangaraju
Zenodo (CERN European Organization for Nuclear Research)
Advanced Neural Network Applications
preprint

When Do Parameter Budgets Misprice CPU Inference? A Matched-Budget Test of Byte-Traffic Pruning Allocation

Prabhu Rangaraju
preprint en

Abstract

Pruning budgets for language models are almost always stated in parameter count, yet CPU decode cost is plausibly driven by the bytes moved per token: weight bytes streamed each step plus KV-cache bytes read at the occupied context. We test, under a pre-registered protocol, whether allocating a 25% structured prune of Llama-3.2-3B by an analytical per-token byte-traffic model beats allocating it by parameter count, at fixed CPU thread caps (2 and 4 threads, both pinned to 2 cores). Seven allocations search one shared unit space (layers, FFN channels, KV groups) and differ only in the cost model used to choose the cut. The result is a bounded negative. In a held-out (leave-one-variant-out) prediction contest, the traffic model does not beat the parameter model: at 2 threads AIC (parameters - traffic) is +2.21 (the traffic model needed >10 to win) and at 4 threads -19.11 (parameters win outright). The predeclared byte allocation is not reliably faster than the parameter allocation to the registered standard: paired time ratios are 0.917 [0.824, 1.020] and 0.986 [0.934, 1.041] at 2 threads (8192 and 16384 context), and 0.818 [0.795, 0.842] and 0.817 [0.609, 1.096] at 4 threads, against a required upper confidence bound below 0.90, and the traffic model's own predicted ratios (0.771, 0.673) are systematically optimistic. Within this matched-budget CPU pruning family, an analytical weight-plus-KV traffic model therefore did not improve held-out prediction or registered selection over parameter count. The negative is about the analytical model, not about traffic-aware pruning as such: pruning by measured latency remains untouched by our result. Two scope limits bind the result: we measure decode speed only (pruned-model quality is not evaluated), and all timing is on a shared 2-core virtual machine whose tenant-load variance — including a documented load-era effect on the earliest cells — is a formal limitation. Descriptively, the byte-greedy pick's mean selection regret is low (3.8% and 1.4% at the two caps, vs 24.4% and 21.3% for the parameter pick) — a good pick, that the model itself does not explain. A post-hoc sensitivity excluding the saturated 16384 context reverses the 4-thread contest, localising the failure at the longest context; the frozen verdict is unchanged. We report the mechanisms consistent with the failure, so the boundary of the traffic model's usefulness is itself the contribution.

Zenodo (CERN European Organization for Nuclear Research)
Advanced Neural Network Applications
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon