DuoDistill: A Dual Knowledge Distillation Framework for Optimizing Top-K Recommender Systems

Knowledge distillation for recommender models is a very challenging task because there can be millions of users and items, and The main focus is on determining the relative order of items based on user preferences, rather than predicting a label as in conventional classification models. Almost all the studies in the literature tackle these challenges by sampling items highly ranked by the teacher model and instructing the student model to mimic the teacher's behavior over the sampled items. However, such an approach may not work well when the gap between the teacher and the student is large, not to mention that learning from a small set of data often leads to overfitting. In this work, we propose a dual knowledge distillation approach that addresses these problems. The proposed knowledge distillation approach consists of two stages: 1) distillation from the teacher model to multiple assistant models of different structures, and 2) distillation from the assistant models to the target student model. We believe that when the assistants have different structures, they can be specialized in distinct areas of the teacher's knowledge space, thereby providing better guidance to the student's learning. We use the CiteULike dataset for our experiments and adopt hit ratio and normalized discounted cumulative gain as performance metrics. The results show that, compared to previous studies, incorporating multiple assistant models with different structures between the teacher and student models significantly enhances the recommendation performance of the student model. These improvements are consistent across all performance metrics used.