Focal Gradient: Improving the Calibration of Deep Neural Networks with an Implicit Loss Function

Deep neural networks used for image classification are typically not well calibrated. Current evidence suggests that this miscalibration is primarily driven by the excessive tendency of the cross-entropy loss to increase the confidence during training. Thus, addressing this undesired behavior arises as a promising approach to achieving deep learning models that are well calibrated by default. In this work, first, we analyze existing log-based loss functions that can be used in place of cross-entropy loss to improve calibration. We show that, despite being built upon cross-entropy loss, existing alternatives lack some sensible properties of cross-entropy for training with gradient descent. Based on these observations, we propose Focal Gradient, an implicit loss function that aims to correct the main driving factor in the miscalibration of the networks while also providing useful properties for training with gradient descent. We conduct a comparative study on image classification benchmarks, considering different factors. The results show that Focal Gradient provides the best calibration performance among all the methods. Additionally, when temperature scaling is included in the picture, Focal Gradient is the only method able to consistently outperform cross-entropy loss. Finally, we also show that some advantages of our proposal can be translated to other related methods.