SQNR-Based Design of 16-Bit Floating-Point Representations for ReLU Activations

This paper proposes an SQNR-based method for the design of bfloat16-inspired 16-bit floating-point representations for ReLU activations. The method exploits the analogy between floating-point formats and piecewise-uniform quantizers and uses SQNR as a performance metric to evaluate alternative bit allocations between the significand and exponent fields for a given input probability density function (PDF). For ReLU activations, the PDF is derived from a Laplacian pre-activation model, showing that ReLU outputs are non-negative and that the sign bit in bfloat16 is redundant. The redundant bit is therefore reallocated within the fixed 16-bit budget, yielding two representations: FP16I, with the bit added to the significand, and FP16II, with the bit added to the exponent. Using the ReLU PDF, closed-form SQNR models are obtained for both representations and used to characterize their performance relative to bfloat16. An analysis over a wide range of input variances shows that FP16I provides a 6.02 dB SQNR improvement over bfloat16, whereas FP16II maintains the bfloat16 SQNR level within a substantially wider variance range. These SQNR models are further applied to guide the selection between FP16I and FP16II for CNN training based on the observed variance range of pre-ReLU activations. It is demonstrated that replacing the bfloat16 ReLU representation with the selected alternative in a bfloat16-quantized CNN results in training performance closer to that of the FP32-based configuration.