RoundGrad: Round and round we go, optimizing quantization as we flow

Deploying deep neural networks on resource-constrained hardware demands aggressive quantization, yet the rounding scheme applied during this process is almost universally chosen for implementation convenience rather than optimized for accuracy. We introduce RoundGrad, a method that treats weight rounding as a first-order optimization problem. Starting from a pretrained model with frozen weights, RoundGrad maintains a learnable score for each weight that determines whether the value rounds up or down; these scores are updated jointly across the entire network via standard backpropagation under a regularized task-loss objective, without layer-wise reconstruction or second-order approximations. Unlike calibration-based post-training quantization methods, RoundGrad optimizes on labeled training data, placing it between traditional PTQ and full quantization-aware training. On ImageNet classification, RoundGrad achieves near-lossless 4-bit weight quantization across standard CNN architectures, using substantially fewer gradient steps than quantization-aware training (up to fewer than standard QAT schedules). Results on Pascal VOC semantic segmentation confirm that the learned rounding policy transfers to dense prediction tasks. The simplicity of the formulation and its competitive accuracy at the deployment-critical 4-bit regime make RoundGrad a practical and principled alternative to existing rounding approaches.