Prompt-driven graph distillation: Enabling single-layer MLPs to outperform Deep Graph Neural Networks in graph-based tasks

Graph distillation endeavors to transfer knowledge from large, complex teacher models, such as Graph Neural Networks (GNNs), to smaller, more efficient student models such as Multi-Layer Perceptrons (MLPs). Our work is motivated by a critical observation: as the complexity and depth of the teacher GNN increase, the performance of the distilled student MLP models tends to decline significantly. This issue is not limited to a specific distillation method but is a prevalent challenge across various GNN to MLP approaches. To address this gap, we introduce a novel prompt-driven graph distillation framework that enhances the student model’s input space by appending or integrating learned prompts with the original features. These prompts, derived from the teacher’s input, hidden and output knowledge, provide additional context that assists the MLP student in distilling information, thereby circumventing the need to directly capture any teacher’s knowledge into the simplest single-layer MLP. Our empirical experiment on benchmark graph datasets reveals a counter-intuitive phenomenon: the more capable the teacher, the greater the student’s ability to utilize prompts to outperform the teacher. This finding highlights the strength of our approach: a well-prompted student can indeed surpass its teacher, at achieving the best performance with an accuracy of 87.2% on the Cora standard split with a single-layer MLP, while also maintaining efficiency and robustness. In addition to testing on benchmark graph datasets, we applied our framework to the TUH EEG Epilepsy Corpus (TUEP), condensing complex GNN models to simple MLPs while achieving good performance and efficiency in epilepsy classification.