A Knowledge Distillation Online Training Circuit for Fault Tolerance in Memristor Crossbar Array-Based Neural Networks

Knowledge distillation is widely used as an effective model compression technique to improve the performance of small models. Most of the current researches on knowledge distillation focus on the algorithmic level and ignore the potential benefits of hardware implementation. In this paper, a multi-loss knowledge distillation online training circuit based on memristor crossbar array is designed, which can improve the inference efficiency and reduce the power consumption of deep learning models on edge devices. The circuit is able to process data in real time, and it can be used to handle stuck-at-faults (SAF) caused by factors such as manufacturing defects in the memristor. Moreover, a fault detection scheme with low time cost is proposed in order to address the low efficiency of stuck-at-fault detection in memristor crossbar arrays. The scheme is combined with a self-compensating pruning method and knowledge distillation online training mechanism, which significantly improves the model training and inference capability of the circuit under fault conditions. Experimental results show that the multi-loss knowledge distillation online training improves the accuracy by 4.15% and 63.48% respectively in two models compared with traditional training schemes. The fault-tolerance scheme reduces the power consumption of the memristor crossbar arrays by 41.2% and 72.6% respectively on the two models, demonstrating its potential and advantages in edge computing.