Efficiency and Stability of a New Hybrid Unconstrained Optimization Algorithm with Quasi-Newton Updates and Higher-Order Methods

We propose the higher-order quasi-Newton (HOQN) method, a hybrid algorithm for unconstrained optimization that combines Newtonian predictors with higher-order correctors derived from vector extensions of the Traub, Chun, and Ostrowski methods, along with quasi-Newton updates of the inverse Hessian using Broyden–Fletcher–Goldfarb–Shanno (BFGS) or Davidon–Fletcher–Powell (DFP) formulas. We demonstrate that the resulting scheme achieves cubic local convergence order, representing a substantial improvement over the superlinear convergence typical of classical quasi-Newton methods, while maintaining a cost of On2 per iteration. We also analyze variants that incorporate two successive quasi-Newton updates, and show that they retain the same cubic order. Numerical experiments with the benchmark functions of Himmelblau and Freudenstein–Roth confirm the theoretical convergence order and show that the hybrid variants consistently require fewer iterations than BFGS, DFP, and Symmetric Rank-One (SR1). In the case of the Booth function, given its strictly convex quadratic structure, the proposed hybrid methods reach the global minimum in just two iterations and exhibit numerical accuracy superior to that of classical quasi-Newton methods. In addition, limited-memory variants (L-HOQN) are introduced; these are evaluated during the training of a convolutional neural network on the MNIST dataset, where they achieve test accuracies exceeding 99% and outperform L-BFGS and standard stochastic gradient descent (SGD) at all tested learning rates.