Swarm-based gradient descent method for non-convex optimization

We introduce a new swarm-based gradient descent (SBGD) method for non-convex optimization. The swarm consists of agents, each is identified with a position, <inline-formula content-type="math/mathml"> <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" alttext="bold x"> <mml:semantics> <mml:mrow class="MJX-TeXAtom-ORD"> <mml:mi mathvariant="bold">x</mml:mi> </mml:mrow> <mml:annotation encoding="application/x-tex">\mathbf {x}</mml:annotation> </mml:semantics> </mml:math> </inline-formula>, and mass, <inline-formula content-type="math/mathml"> <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" alttext="m"> <mml:semantics> <mml:mi>m</mml:mi> <mml:annotation encoding="application/x-tex">m</mml:annotation> </mml:semantics> </mml:math> </inline-formula>. The key to their dynamics is communication: masses are being transferred from agents at high ground to low(-est) ground. At the same time, agents change positions with step size adjusted to their relative mass. Accordingly, the crowd of agents is dynamically divided between heavier ‘leaders’, which proceed with small time steps and are expected to approach local minima, and lighter ‘explorers’ which proceed with larger-step protocol and are expected to encounter improved position for the swarm. If they do, then they assume the role of heavy swarm leaders and so on. We prove local convergence and we present convincing numerical simulations which demonstrate that the added layer of SBGD explorers performs well in improving its global convergence behavior in one-, two-, and 20-dimensional benchmarks.