A Distributed Particle Swarm Optimization Algorithm Based on Apache Spark for Asynchronous Parallel Training of Deep Neural Networks

Deep Learning Neural Networks (DLNN) require an immense amount of computation, especially in the training phase when multiple layers of intermediate neurons need to be built. The situation is even more dramatic today with the proliferation of applications with intelligence at the edge, not just in the cloud. Therefore, to meet the new requirements of edge computing, it is imperative to accelerate the execution phases of neural networks as much as possible. In this paper, we will focus on the algorithm known as Particle Swarm Optimization (PSO). It is a bio-inspired, stochastic optimization approach whose goal is to iteratively improve the solution to a given (usually complex) problem by attempting to approximate a given objective. The use of PSO in an edge computing environment has the potential to make the training of the DLNN there without the need to transfer resource-intensive tasks to the cloud. However, implementing an efficient PSO is not a straightforward process due to the complexity of the computations performed on the swarm of particles and the iterative execution until until a near-target solution with minimal error is achieved. In the present work, two parallelizations of the PSO algorithm have been implemented, both designed for a distributed execution environment (Apache Spark). The first PSO parallelization follows a synchronous scheme; i.e., the best global position found by particles is globally updated before the execution of the next iteration of the algorithm. This implementation proved to be more efficient for medium-sized datasets (< 40000 data points). In contrast, the second implementation is an asynchronous parallel variant of the PSO algorithm, which showed lower execution time for large datasets (> 170,000 data points) compared to the first one. Additionally, it exhibits better scalability and elasticity with respect to increasing dataset size. Both variants of the PSO have been implemented to distribute the computational load (particle fitness evaluation and position update) across the different executor nodes of the Spark cluster to effectively achieve a coarse-grained parallelism, resulting in a significant performance imporvement over the current sequential versions of the PSO.