Eliminating Harmonic Quantization Artifacts in Deep Neural Networks via Triangular Probability Density Function (TPDF) Dithering

Post-Training Quantization (PTQ) to sub-8-bit representations (INT6, INT4) is critical for reducing the memory footprint and compute intensity of large-scale Transformer architectures. However, conventional uniform quantization almost universally employs deterministic Round-to-Nearest (RTN) rounding. While high-precision regimes permit treating quantization error as benign additive white Gaussian noise, this assumption catastrophically breaks down below 8 bits. In sub-8-bit regimes, deterministic RTN generates non-linear, signal-correlated harmonic distortion that coherently accumulates across deep residual blocks and exponentiates within Softmax attention operators, triggering attention entropy degradation and singular spectrum distortion. In this work, we present a rigorous theoretical foundation and empirical framework for non-subtractive Triangular Probability Density Function (TPDF) dithering in deep neural network tensor operations. By analyzing the Fourier expansion of the quantizer error characteristic and applying Schuchman's condition within the non-overload granular region, we provide complete analytical proofs that: (1) injecting 2-LSB peak-to-peak TPDF dither guarantees an identically zero conditional error expectation (E[e|w] = 0, for all w), rendering quantized General Matrix Multiplications (GEMM) unbiased estimators of full precision and eliminating first-order O(Delta) task-loss gradient bias; and (2) TPDF dither establishes an identically constant conditional error variance (Var[e|w] = Delta^2/4, for all w) by virtue of the double zeros of the sinc^2 characteristic function at harmonic frequencies, completely eliminating noise power modulation. In contrast, Uniform (RPDF) dither suffers from persistent signal-dependent variance ripple of 0.25*Delta^2. We further establish exact non-saturation guard-band conditions, analyze static frozen versus dynamic runtime dither regimes, and prove orientation-independent angular preservation in self-attention. Empirically, TPDF dithering improves Spurious-Free Dynamic Range (SFDR) by +16.6 dBFS over RTN at INT4, eliminates systematic input-error cross-correlation (reducing empirical rho from -0.0882 to -0.0050, statistically indistinguishable from theoretical zero at N=32,768), reduces 32-layer systematic directional bias drift by 63.0%, and prevents attention entropy collapse in Transformer blocks.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-14
DOI
https://doi.org/10.5281/zenodo.24218729
Primary Topic
Advanced Neural Network Applications
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Eliminating Harmonic Quantization Artifacts in Deep Neural Networks via Triangular Probability Density Function (TPDF) Dithering

Yaroslav Aleksandrovich Tarkov
Zenodo (CERN European Organization for Nuclear Research)
Advanced Neural Network Applications
preprint

Eliminating Harmonic Quantization Artifacts in Deep Neural Networks via Triangular Probability Density Function (TPDF) Dithering

Yaroslav Aleksandrovich Tarkov
preprint en

Abstract

Post-Training Quantization (PTQ) to sub-8-bit representations (INT6, INT4) is critical for reducing the memory footprint and compute intensity of large-scale Transformer architectures. However, conventional uniform quantization almost universally employs deterministic Round-to-Nearest (RTN) rounding. While high-precision regimes permit treating quantization error as benign additive white Gaussian noise, this assumption catastrophically breaks down below 8 bits. In sub-8-bit regimes, deterministic RTN generates non-linear, signal-correlated harmonic distortion that coherently accumulates across deep residual blocks and exponentiates within Softmax attention operators, triggering attention entropy degradation and singular spectrum distortion. In this work, we present a rigorous theoretical foundation and empirical framework for non-subtractive Triangular Probability Density Function (TPDF) dithering in deep neural network tensor operations. By analyzing the Fourier expansion of the quantizer error characteristic and applying Schuchman's condition within the non-overload granular region, we provide complete analytical proofs that: (1) injecting 2-LSB peak-to-peak TPDF dither guarantees an identically zero conditional error expectation (E[e|w] = 0, for all w), rendering quantized General Matrix Multiplications (GEMM) unbiased estimators of full precision and eliminating first-order O(Delta) task-loss gradient bias; and (2) TPDF dither establishes an identically constant conditional error variance (Var[e|w] = Delta^2/4, for all w) by virtue of the double zeros of the sinc^2 characteristic function at harmonic frequencies, completely eliminating noise power modulation. In contrast, Uniform (RPDF) dither suffers from persistent signal-dependent variance ripple of 0.25*Delta^2. We further establish exact non-saturation guard-band conditions, analyze static frozen versus dynamic runtime dither regimes, and prove orientation-independent angular preservation in self-attention. Empirically, TPDF dithering improves Spurious-Free Dynamic Range (SFDR) by +16.6 dBFS over RTN at INT4, eliminates systematic input-error cross-correlation (reducing empirical rho from -0.0882 to -0.0050, statistically indistinguishable from theoretical zero at N=32,768), reduces 32-layer systematic directional bias drift by 63.0%, and prevents attention entropy collapse in Transformer blocks.

Zenodo (CERN European Organization for Nuclear Research)
Advanced Neural Network Applications
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon