Bundle Adjustment With Backtracking Line Search on Manifold

Bundle adjustment (BA) is a fundamental problem in visual 3D reconstruction. The Levenberg-Marquardt (LM) algorithm, a trust region method, is widely regarded as the gold standard for solving BA problems. In each LM iteration, the current solution is updated by an increment vector derived from solving a linear system with a damping factor to regularize the step size. However, directly applying this increment may fail to reduce the reprojection cost. To address this problem, the LM algorithm employs a trial-and-error strategy. Specifically, it repeatedly solves the linear system with an increasing damping factor until the cost decreases. This process leads to invalid iterations. Since solving the linear system is typically the most time-consuming step and a large damping factor limits the step size in the subsequent iterations, this strategy wastes computational resources and slows down convergence. However, this issue has received little attention in prior research on BA. On the other hand, line search offers an alternative technique to control the step size, however, its application to BA remains underexplored. This paper presents a simple yet effective solution to overcome the limitation of the LM algorithm. We introduce on-manifold backtracking line search into the LM algorithm to accelerate convergence. The Armijo condition is adopted to ensure a sufficient decrease in reprojection cost. We show that the Armijo condition on manifold can be efficiently computed in the LM framework. By fusing line search and the LM algorithm to control the step size, our method effectively reduces the number of invalid iterations and improves convergence speed. Extensive empirical evaluations on both unstructured internet image collections and sequential image streams show that our algorithm converges significantly faster compared to state-of-the-art BA algorithms.