Quantifying Student Model Accuracy via Sample Size and Soft Label MSE: Insights into Label Quality

Knowledge distillation has received significant attention due to its ability to improve model efficiency without significant performance degradation. However, there is still a research gap in understanding the relative impact of various metrics used to evaluate the influence of soft labels on student model performance. This study aims to quantify student model accuracy through sample size and the soft label’s mean squared error (MSE). We use a dataset from our previous study called the classification-integrated moving average (CIMA) for knowledge distillation training. We evaluate soft labels from deep neural networks (DNN), gradient boosting, naïve Bayes, and support vector machine (SVM) methods as teacher models. We use k-nearest neighbor (KNN) regression as a student model. We then leverage some underfitted DNN to control the quality of soft labels, observing and quantifying the performance of the student model based on the number of samples and soft label quality. Our final novel equation has r^2 and root MSE (RMSE) of 0.86 and 0.003, respectively. Future works can investigate the generalization of this equation, which can lead to the exploration of new parameters and broader case studies.