Real Field Error Correction for Coded Distributed Computing-Based Training

This work targets at real field error corrections in coded distributed computing (CDC) based deep learning (DL). On one hand, shift-and-addition (SA) encoding and zigzag decoding (ZD) is adopted in the encoding and decoding process, respectively, which collectively aim at reaping low computational complexity. On the other hand, storing real number induces truncation error, which effect becomes prominent under SA operation, and the truncation error may magnify and spread to a wide area along the ZD decoding process. These errors may temporarily drag the iteration point away from its working route which is equivalent to slowing down the training process. Firstly, to alleviate the truncation error magnification effect, we propose CDC-Pre/Post framework which pre-treats the data before encoding and post-treats the data after decoding. Secondly, to correct errors, we propose a method that uses convolution code to re-interprete the SA encoding structure, and design largest cluster decoding (LCD) algorithm. Thirdly, to enjoy both fast ZD and error correction capability of LCD, we propose smooth transition from ZD to LCD (ST-ZD-LCD) which is incremental with newly received packet. Experimental results show advantage of our scheme in two fold: First, it achieves significantly better error correction performance than naive ZD. Second, the training time of ST-ZD-LCD based distributed DL system is shortened significantly.