Accelerating the Convergence of Poorly-Scaled Optimization Problems

Poorly-scaled optimization problems have ill-conditioned Hessians or Jacobians, which can lead to slow convergence for many optimization algorithms. Indeed, if the ill-conditioning is severe enough, the optimization algorithm may not converge before the available computational budget is exhausted. To help mitigate scaling issues, we propose mappings for the design parameters and constraints that are based on the eigenvalue decomposition of the Hessian and singular-value decomposition of the Jacobian, respectively. To avoid the expense of computing the full decompositions, the mappings rely on only a small subset corresponding to the eigenvalues and singular values with the largest magnitudes. We estimate the eigenpairs using Arnoldi's method with Hessian-vector products approximated using forward differences applied to the gradient, which avoids the need for second derivatives. Similarly, we estimate the largest singular values of the constraint Jacobian using a matrix-free Lanczos algorithm. We demonstrate the Hessian-based mapping using unconstrained problems with quadratic objectives defined over a range of design-space dimensions and Hessian condition numbers. These numerical experiments demonstrate that the Hessian-based mapping can help reduce the number of gradient evaluations by 3 to 12 times for poorly-scaled quadratic objectives. We demonstrate the Hessian- and Jacobian-based mappings with linearly-constrained quadratic programs. For constrained problems, Hessian-based scaling also accelerates the convergence of poorly scaled problems; however, the Jacobian scaling provides no benefits for the sequential quadratic programming (SQP) optimization algorithm considered. The Hessian-based mapping can accelerate the convergence of existing optimization software libraries without requiring changes to either the optimizer or the engineering analysis code.