Preserving node representation diversity in deep graph neural networks through Rényi entropy regularization

Abstract Graph Neural Networks have demonstrated remarkable performance across diverse domains, yet they face significant limitations in deeper architectures due to oversmoothing—the phenomenon where node representations become increasingly indistinguishable through successive message-passing operations. We propose a novel information-theoretic approach that addresses this fundamental challenge through Rényi entropy optimization. Our method quantifies and maximizes the diversity of node representations across network layers using kernel density estimation with Gaussian kernels. By formulating a graph-structured entropy regularization term that respects the underlying topology, we encourage networks to maintain discriminative features while preserving essential structural information. This approach integrates seamlessly with existing GNN architectures, requiring minimal modifications to the training procedure. Extensive experiments on ten benchmark datasets demonstrate that our Rényi entropy regularization consistently improves performance across multiple GNN variants, with average gains of 1.89% and particularly significant improvements on heterophilic graphs exceeding 2.5%. Our depth analysis shows that the method extends viable network depth from 2–3 layers to 8–10 layers, effectively countering the homogenization tendency of deep GNNs and establishing a principled foundation for more expressive graph neural networks.