How Many Bits Does a Model Weight Need? The Delta-Squared-Over-Twelve Law for Quantised Regression Weights, Tested on Six UCI Data Sets With Nearest Rounding, Stochastic Rounding and Subtractive Dither

Storing model weights in fewer bits saves memory and energy, and the classical model of quantisation noise says what it costs: rounding to a grid of step Delta adds an error of variance Delta^2/12 to each weight, so the expected increase in squared prediction error is Delta^2/12 times the trace of the feature second-moment matrix, and each extra bit cuts it by a factor of 4. I test this on random-Fourier-feature ridge regressions with 2,048 weights fitted to six public UCI data sets, quantising the weights to 2 to 12 bits with three rules. Subtractive dither makes the law exact for any weights, and I prove it; across bit widths from 2 to 12 the mean over data sets of its quadratic term is between 0.88 and 1.10 times the prediction. Stochastic rounding is unbiased but doubles the error variance, which I prove for uniformly placed weights and observe (1.99 to 2.03 times Delta^2/12 from 4 bits up). Ordinary nearest rounding matches the law from about 5 bits but is 6 to 25 times worse at 2 to 4 bits, because the rounding error then copies the sign pattern of the weights. A formula for the bits needed to keep the test error increase within 1% matches dither at all six data sets and nearest rounding to within 2 bits (4 to 10 bits predicted). Two further results explain why single quantised models scatter around the law: the quadratic term's relative standard deviation is sqrt(2 tr(H^2) - 1.2 sum H_jj^2)/tr(H), about 0.9 here because the effective rank of the features is only about 2.5, and the test-set gradient adds a cross term that shrinks only by a factor 2 per bit and overtakes the quadratic term at about 8 bits.