Interpolasyon Tabanlı Paralel Öğrenme Üzerine İncelemeler Investigations on Interpolation Based Distributed Learning

Artificial neural networks start their training at a random point in their complex parameter loss space. They use gradient descent based algorithms to optimize the error on the data set they are trained on. At the end of training, they reach a point (local minimum) where the error on the trained data is small. Due to the non-convex nature of the space, different initial points reach different local minima. It is known that the loss value of the points obtained when linear interpolation is performed between these points is high. In addition, it has also been observed in the literature that this situation is not observed between models trained in parallel with different data starting from a common training point and that successful results can be obtained when training is continued by fine-tuning from the best point between them. However, this approach used a single dataset and a single model. In this study, in order to test the consistency of this approach, training was performed with different architectures and different datasets. We also examined the effect of the size of the data used for common training and the number of parallel training processes. The results show that the proposed approach is successful regardless of the dataset, architecture, common training set ratio and number of parallel processes.