Double‐Integration‐Enhanced Stochastic Gradient Descent Based on Neural Dynamics for Improving Generalisation

ABSTRACT Generalisation is a crucial aspect of deep learning, enabling models to perform well on unseen data. Currently, most optimisers that improve generalisation typically suffer from efficiency bottlenecks. This paper proposes a double‐integration‐enhanced stochastic gradient descent (DIESGD) optimiser, which treats the negative gradient as an error term and the parameter's rate of change as a noise term. The generalisation superiority of DIESGD originates from its parameter update mechanism, which inherently favours flat minima. The convergence of DIESGD is proven, and extensive experiments are conducted on a variety of datasets and models, demonstrating the superior performance of DIESGD compared to existing optimisers. Furthermore, DIESGD exhibits reduced sensitivity to the initial learning rate, which enhances its robustness and ease of use in practical applications. The proposed DIESGD optimiser connects the distinct fields of optimisers in deep learning and neural dynamics, which is a typical form of dynamic neural networks. The source code is available at https://github.com/LongJin‐lab/DIESGD .