Incorporating Knowledge Distillation and Compound Scaling to Improve the EfficientNet Model Performance

A simpler student model that performs as well as a robust teacher model on the inference server can offer many advantages. The crucial advantages are computational efficiency and effectiveness. It can be achieved using the knowledge distillation (KD) technique, which distills knowledge from robust models to weaker ones. In this study, we investigated optimizing the performance of the EfficientNet model by combining the KD technique with the compound scaling concept. We tried to train the EfficientNet student model with a one-level-higher teacher model. Using our method, student model performance improves compared to the standard transfer learning (STL) method. In addition, we found that the optimal percentage split between student and teacher models is 50% student model and 50% teacher model. All EfficientNet models improved their performance with this composition compared to the STL method. We also prove whether compound scaling in the EfficientNet model supports the KD technique in enhancing performance. The result is that student models tend to perform better when assisted by EfficientNet peers. Finally, some student models taught with our proposed method can outperform the teacher models trained with the STL method. EfficientNet B4, B5, and B6 models trained using our proposed method successfully outperformed the one level higher EfficientNet models trained with the STL method. Specifically, these student models achieved peak accuracies of 92.90%, 93.31%, and 93.86%, surpassing their respective teacher models’ accuracies of 92.63%, 93.02%, and 93.18%.