Data-Free Knowledge Distillation via Contrastive Inversion and Cluster Alignment

Data-free knowledge distillation (DFKD) aims to transfer the knowledge from a large teacher network to a lightweight student network without accessing the original training data. However, existing DFKD methods face the issue of distribution shift during data synthesis, which leads to a decline in knowledge distillation. To address this issue, we propose a data-free knowledge distillation method (CCD) based on contrastive inversion (CI) and cluster alignment (CA). Specifically, CI enables the generative network to better capture relevant features and efficiently synthesize images that conforms to the original training data. However, using CI alone can severely reduce the diversity of synthesis images, which may lead to poor performance of the student network. Thus, we propose CA to assure the generative network can learn not only intra-class feature but also the inter-class feature to alleviate the diversity issue. The experiments on various dataset show that our proposed method outperform other baseline methods.