Learning Label Perturbations

Supervised learning typically uses hard labels for annotations, which may not fully capture the underlying distribution of the data. In the literature, label smoothing is a method that can reduce overconfidence and enhance the model generalization by using a weighted average of one-hot vectors and the uniform distribution, but it does not extract the intrinsic information from the data. Another approach is knowledge distillation, which uses the predicted probability distribution of a teacher network trained with hard labels as soft labels for a student network. However, this method lacks a theoretical explanation. In this work, we draw inspiration from the influence function and propose a post-hoc label perturbation learning method called Deep Soft Label Learning (DSLL). This method iteratively leverages the inherent information present in both the model and data to theoretically determine optimal labels for classification and regression problems. Our experiments demonstrate that DSLL consistently enhances model performance across various tasks, including image classification and object detection.