Expediting Convergence via Polling Optimisation for Gradient Descent in Neural Networks

Optimising the learning rate is essential for efficient neural network training, but static methods can cause overshooting or undershooting, while adaptive techniques like ADAM often struggle to balance exploration and exploitation. We introduce the Polling Method, an ensemble-based optimisation approach that dynamically selects the most effective learning rate at each step, improving convergence and mitigating issues inherent in traditional optimisation strategies. By evaluating base models with varying learning rates at each epoch, the method adaptively balances exploration and exploitation without being constrained by predefined functions or gradient noise. This study details the theoretical foundation, implementation, and integration of the Polling Method with the ADAM optimiser, demonstrating its effectiveness in Artificial Neural Networks and Bayesian variational inference. The results demonstrate that Polling Method-ADAM reduces absolute error by 50% compared to ADAM alone, while also accelerating convergence. In Bayesian optimisation, it reduces the mean gradient shift from 0.85 to 0.7835 over 500 iterations, indicating improved stability in high-dimensional problems. By introducing adaptive learning rate selection within training, the Polling Method enhances optimisation efficiency while mitigating noise accumulation. This framework provides a computationally efficient, flexible alternative for deep learning applications, offering significant improvements over traditional optimisers and a potential breakthrough in neural network training strategies.