GraDiNet: Implicit Self-Distillation of Graph Structural Knowledge

Graph Knowledge Distillation (GKD) in artificial intelligence typically employs a teacher-student model, which faces challenges such as rigidity, time-consumption, and teacher training. To improve, this paper develops a Graph self-Distillation Network (GraDiNet), a framework that operates without the need for a teacher model or graph neural network (GNN) during training and inferencing phases. GraDiNet uniquely utilizes multi-layer perceptrons (MLPs) to harness both the structural knowledge of graphs and the semantic information of nodes, thus facilitating hierarchical self-distillation between a target node and its neighbors. Additionally, the GraDiNet approach incorporates a novel similarity-based difference enhancement technique and a penalty factor within the training loss to further delineate the distinction between positive and negative samples. This allows GraDiNet not only to bypass the necessity for a GNN teacher in learning graph structure knowledge but also to predict node classification efficiently. Extensive evaluations show that standard MLPs can significantly boost their performance through this implicit hierarchical self-distillation and the similarity difference enhancement. GraDiNet thus achieves an average improvement of 15% over conventional MLPs and outperforms leading state-of-the-art GKD methods across three real-world datasets.