Enhancing Neural Network Performance on Tabular Data via Knowledge Distillation and RankGauss Transformation

This study explores how knowledge distillation can effectively transfer the predictive power of gradient boosting models to compact neural networks for tabular data. We propose a streamlined training framework that unifies soft-label supervision from a LightGBM teacher with hard-label learning using either binary cross-entropy or Focal Loss. To improve training stability and generalization, we incorporate RankGauss normalization and temperature scaling. Extensive experiments on real-world datasets show that our approach significantly improves student model performance, increasing the validation AUC from 0.8888 (baseline) to 0.9033 in the best configuration. These findings suggest that, with carefully crafted distillation strategies, even simple feedforward neural networks can approach the accuracy of large ensemble models while achieving better inference efficiency.