Boosting MLPs on Graphs via Distillation in Resource-Constrained Environment

Graph Neural Networks (GNNs) have emerged as a powerful technique across various applications, due to their effective message-passing mechanism. However, their deployment is constrained by limited computational resources, energy concerns, and low-latency processing requirements. While existing works employ logit-based knowledge distillation from GNNs to guide Multilayer Perceptrons (MLPs) training, these methods may lead to reduced accuracy and compromised robustness. These drawbacks arise from two primary factors: the insufficient exploitation of the rich information embedded within the graph structures and the inherent susceptibility of MLPs to noisy data. To tackle these issues, we propose a Mixed Multi-order Knowledge Distillation (MMKD) method, which combines the GNN's logits with hidden layer information through the multi-order distillation to improve the accuracy of the MLP. Moreover, we employ both raw data and perturbed data as input, enhancing the density of knowledge extraction as well as the MLPs' generalization. Extensive experiments across seven benchmark datasets verify the superior performance of our approach in terms of effectiveness and robustness. In comparison with the baseline, our approach achieves an accuracy improvement of up to 8.68% in typical GNN tasks.