Empirical Scaling Laws of Dither Noise Suppression in Foundation Models: From Sub-1B to 27B Parameters Under Quasi-Subtractive TPDF Quantization

Post-training quantization (PTQ) of Large Language Models (LLMs) to sub-8-bit representations introduces severe systematic distortion when performed via deterministic Round-to-Nearest (RTN). Although classical signal processing establishes that Triangular Probability Density Function (TPDF) dithering strictly linearizes quantizers and eliminates harmonic quantization spurs, early experiments on sub-billion parameter models reported degraded perplexity due to additive dither variance (Delta^2 / 4). In this work, we reveal and rigorously prove the fundamental Scaling Law of Dither Noise Suppression: across layer projection dimension D, Central Limit Theorem (CLT) noise averaging suppresses additive dither variance as O(D^{-1/2}), while deterministic RTN bias accumulates coherently across depth L as O(L). Across an empirical evaluation ladder of modern hybrid State-Space Model (SSM) and Transformer foundation models (the Qwen3.5/Qwen3.8 family spanning D=1024 to D=5120, L=24 to L=64, and 0.8B to 27B parameters), we demonstrate that for D >= 4096, non-subtractive TPDF inherently outperforms RTN in directional representation preservation. Furthermore, we invent Quasi-Subtractive TPDF Quantization, an out-of-core architectural mechanism that dynamically reconstructs pseudo-random dither sequences in compute registers from deterministic counter-based PRNG seeds (Philox-2x32), algebraically canceling dither variance back to the theoretical minimum of Delta^2 / 12 without caching full-precision weights. In Qwen3.5-9B, Quasi-Subtractive TPDF achieves an unprecedented 95.8% reduction in cumulative directional drift relative to RTN (8.97e-6 vs. 2.13e-4). On the flagship 64-layer Qwen3.8-27B model executed via layer-by-layer out-of-core streaming within 4.2 GB VRAM on consumer hardware, Quasi-Subtractive TPDF preserves a cosine fidelity of 0.997852 against uncompressed FP16. Finally, we demonstrate that TPDF provides dual auxiliary benefits: enabling 4.0x long-context Key-Value (KV) cache compression with zero attention entropy degradation up to 4096 tokens, and inducing intrinsic stochastic gradient masking that diminishes adversarial white-box jailbreak gradient alignment from 0.9920 to 0.9674.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-15
DOI
https://doi.org/10.5281/zenodo.24247444
Primary Topic
Advanced Neural Network Applications
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Empirical Scaling Laws of Dither Noise Suppression in Foundation Models: From Sub-1B to 27B Parameters Under Quasi-Subtractive TPDF Quantization

Yaroslav Aleksandrovich Tarkov
Zenodo (CERN European Organization for Nuclear Research)
Advanced Neural Network Applications
preprint

Empirical Scaling Laws of Dither Noise Suppression in Foundation Models: From Sub-1B to 27B Parameters Under Quasi-Subtractive TPDF Quantization

Yaroslav Aleksandrovich Tarkov
preprint en

Abstract

Post-training quantization (PTQ) of Large Language Models (LLMs) to sub-8-bit representations introduces severe systematic distortion when performed via deterministic Round-to-Nearest (RTN). Although classical signal processing establishes that Triangular Probability Density Function (TPDF) dithering strictly linearizes quantizers and eliminates harmonic quantization spurs, early experiments on sub-billion parameter models reported degraded perplexity due to additive dither variance (Delta^2 / 4). In this work, we reveal and rigorously prove the fundamental Scaling Law of Dither Noise Suppression: across layer projection dimension D, Central Limit Theorem (CLT) noise averaging suppresses additive dither variance as O(D^{-1/2}), while deterministic RTN bias accumulates coherently across depth L as O(L). Across an empirical evaluation ladder of modern hybrid State-Space Model (SSM) and Transformer foundation models (the Qwen3.5/Qwen3.8 family spanning D=1024 to D=5120, L=24 to L=64, and 0.8B to 27B parameters), we demonstrate that for D >= 4096, non-subtractive TPDF inherently outperforms RTN in directional representation preservation. Furthermore, we invent Quasi-Subtractive TPDF Quantization, an out-of-core architectural mechanism that dynamically reconstructs pseudo-random dither sequences in compute registers from deterministic counter-based PRNG seeds (Philox-2x32), algebraically canceling dither variance back to the theoretical minimum of Delta^2 / 12 without caching full-precision weights. In Qwen3.5-9B, Quasi-Subtractive TPDF achieves an unprecedented 95.8% reduction in cumulative directional drift relative to RTN (8.97e-6 vs. 2.13e-4). On the flagship 64-layer Qwen3.8-27B model executed via layer-by-layer out-of-core streaming within 4.2 GB VRAM on consumer hardware, Quasi-Subtractive TPDF preserves a cosine fidelity of 0.997852 against uncompressed FP16. Finally, we demonstrate that TPDF provides dual auxiliary benefits: enabling 4.0x long-context Key-Value (KV) cache compression with zero attention entropy degradation up to 4096 tokens, and inducing intrinsic stochastic gradient masking that diminishes adversarial white-box jailbreak gradient alignment from 0.9920 to 0.9674.

Zenodo (CERN European Organization for Nuclear Research)
Advanced Neural Network Applications
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon
Empirical Scaling Laws of Dither Noise Suppression in Foundation Models: From Sub-1B to 27B Parameters Under Quasi-Subtractive TPDF Quantization — Yaroslav Aleksandrovich Tarkov · Zenodo (CERN European Organization for Nuclear Research) (2026) | TGRS Research Map | TGRS