Application of Knowledge Distilled Neural Network to Railway Overhead Line Fault Diagnosis

This research has applied the teacher-student model of knowledge distillation (KD) technique to compress artificial neural networks (ANN) and Densely Connected Convolutional Networks (DenseNet) for efficiently examining railway overhead line's footage and diagnosing its operational health. The feature extraction process is optimized by a Satisfied Learning Transfer (STL) strategy; the KD+STL combination refines the ‘dark knowledge’ in the teacher model and updates the student model through a proactive feedback mechanism, leading to higher diagnostic accuracy. Furthermore, to keep the model simple with a practicable number of parameters while minimizing the loss of performance, the benefits of adopting differential KD and attention-based probabilistic factor knowledge distillation assisted KD (APF-KD) are discussed. The footage is segmented and compressed, and the results of different KDs algorithms are presented for comparison and recommendations.