Variable Projection algorithms: Theoretical insights and a novel approach for problems with large residual

This paper delves into an in-depth exploration of the Variable Projection (VP) algorithm, a powerful tool for solving separable nonlinear optimization problems across multiple domains, including system identification , image processing , and machine learning. We first establish a theoretical framework to examine the effect of the approximate treatment of the coupling relationship among parameters on the local convergence of the VP algorithm and theoretically prove that the Kaufman’s VP algorithm can achieve a similar convergence rate as the Golub & Pereyra’s form. These studies fill the gap in the existing convergence theory analysis, and provide a solid foundation for understanding the mechanism of VP algorithm and broadening its application horizons. Furthermore, inspired by these theoretical insights, we design a refined VP algorithm, termed VPLR, to address separable nonlinear optimization problems with large residual. This algorithm enhances convergence performance by addressing the coupling relationship between parameters in separable models and continually refining the approximated Hessian matrix to counteract the influence of large residual. The effectiveness of this refined algorithm is corroborated through numerical experiments.