Fault Tolerant Data and Model Parallel Deep Learning in Edge Computing Networks

Deep learning (DL) training or retraining on an edge computing network is promising due to its local computation advantage over the cloud. Data and model parallel DL training is a solution to handle the challenges of the large and increasing scale of deep neural network (DNN) models and resource constraints of edge devices. However, its training accuracy by a certain time will be greatly affected or the training process will slow down if one model partition experiences faults due to causes such as edge node mobility, resource unavailability and hardware errors. Though task replication strategies have been used in wireless networks or the TensorFlow machine learning framework, they cannot handle DL job unique features (i.e., large amounts of intermediate results, high accuracy requirement and DL job dataflow) in such DL training. To address this problem, we propose a proactive Fault Tolerant data and model parallel DL system at the Edge (FTLE) to improve DL training time and accuracy by a certain time. FTLE proactively calculates and predicts the fault probability of each model partition using a DL method. It then decides the number of replicas of each partition based on its fault probability, its significance on the accuracy and its importance in the DL job dataflow. Then, FTLE assigns the replicas to edge devices, and schedules a replica to run before each predicted fault occurs based on the DL job dataflow in order to reduce training time and replication time overhead. Our trace-driven experiment on real devices shows that for one epoch training, FTLE shows around 59% reduction in training time, 45% increase in accuracy, and 65% reduction in replication overhead compared to other methods.