Teacher Fine-Tuning-Based Quantization Distillation

The methods of knowledge distillation (KD) and quantization are distinct approaches that serve the purpose of reducing the computational complexity and memory requirements of a given model. The concept of KD is founded upon the basic idea that guided learning is better than independent learning. KD involves training a smaller model, known as student, under the guidance of a larger model, known as teacher, with the disadvantage of requiring the pre-training of the larger model. The process of quantization entails the conversion of floating-point parameters, which are commonly utilized, into integer parameters. This conversion results in a substantial reduction in the storage requirements of deep models and an increase in the speed of inference with the drawback of performance loss. This paper presents an effective combination of the approaches above, named Teacher Fine-Tuning-Based Quantization Distillation (TFQD) to integrate KD into quantization aware training. Specifically, we use the full-precision model as the teacher and the quantized model as the student for “self-distillation.” It effectively reduces the accuracy degradation resulting from quantization without needing pre-training on a larger model. In addition, we update the teacher during the distillation process with the idea of meta-learning. We further suggest a linearly correlated constraint strategy to alleviate the adverse effects of a strict constraint, given by the Kullback-Leibler divergence, on student performance in conventional KD. We show that TFQD outperformed existing state-of-the-art methods in multiple experiments on the CIFAR100 and Tiny-ImageNet datasets.