EKD4Rec: Ensemble Knowledge Distillation from LLM-based Models to Traditional Sequential Recommenders

In this paper, we propose a new ensemble knowledge distillation method for distilling knowledge from LLM-based recommendation (teacher) models to traditional light-weight sequential recommendation (student) models. In particular, instead of using one single teacher model, the averaged prediction from multiple teachers is employed as the soft targets of knowledge distillation. Further, only the top K soft labels of teachers' output distribution are sampled for distillation to make it more focused on the corresponding high-ranked items. Extensive experiments on three public datasets show the effectiveness of the proposed ensemble knowledge distillation for sequential recommendation (EKD4Rec).