Residual Learning for Self-Knowledge Distillation: Enhancing Neural Networks Through Consistency Across Layers

Knowledge distillation is widely used technique to transfer knowledge from a large pretrained teacher network to a small student network. However, training complex teacher models requires significant computational resources and storage. To address this, a growing area of research, known as self-knowledge distillation (Self-KD), aims to enhance the performance of a neural network by leveraging its own latent knowledge. Despite its potential, existing Self-KD methods often struggle to effectively extract and utilize the model's dark knowledge. In this work, we identify a consistency problem between feature layer and output layer, and propose a novel Self-KD approach called <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">R</b>esidual Learning for <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">S</b>elf-<bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">K</b>nowledge <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">D</b>istillation (<bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">RSKD</b>). Our method addresses this issue by enabling the last feature layer of the student model learn the residual gap between the outputs of the pseudo-teacher and the student. Additionally, we extend RSKD by allowing each intermediate feature layer of the student model to learn the residual gap between the corresponding deeper features of the pseudo-teacher and the student. Extensive experiments on various visual datasets demonstrate the effectiveness of the proposed method, which outperforms the state-of-the-art baselines.