Ternary Logit Distillation via Non-Target Classes Decomposition

Knowledge Distillation (KD) is an effective model compression approach to transfer knowledge from a larger network to a smaller one. Existing state-of-the-art methods mostly focus on feature-based knowledge, however, due to the ease of implementation and the privacy of transferred knowledge, logit distillation has its irreplaceability. In this paper, we redirect our attention to exploring the underutilized potential inherent in logit-based knowledge. We propose a method named TLD (Ternary Logit Distillation), which transferring knowledge through three distillation components derived from decouple and reform the logit outputs. To be specific, we introduce a confidence-based method to decouple the logit outputs and reformulate the classical logit distillation loss into a weighted sum of three distinct distillation components: TCKD (Three-way Classification Knowledge Distillation), SCKD (Similar Classification Knowledge Distillation), and DCKD (Dissimilar Classification Knowledge Distillation). We explore the significance of each distillation component and reduces the inhibitory interactions among the three components through a redistribution of weights and temperature hyper-parameter, thereby creating a logit based distillation method with superior performance and low computation load. We have conducted extensive experiments on classification models across a range of architectures such as ResNet, VGG, and MobileNet, and diverse datasets including CIFAR100 and ImageNet. All of them witness our proposed method achieving state-of-the-art performance in comparisons with other baseline techniques based on logit and feature knowledge.