Neural Network Parameter Synchronization Using Distributed Source Coding Principles

Neural networks have helped solve numerous problems and create applications in various fields in the past few years. In order for the neural networks to perform well, the number of parameters has had to grow considerably in size. One problem that this introduces is when one fine-tunes the parameters, a large amount of bandwidth is needed to synchronize other users to use the same parameters. In this paper, we propose a synchronization protocol that uses Distributed Source Coding principles to significantly reduce the bandwidth needed to synchronize these parameters. The protocol incorporates a tree-based quantizer in conjunction with CRC codes. We further demonstrate how LT codes can be used to save bandwidth when a sender broadcasts the parameters to many users.