On the Convergence Rates of Iterative Regularization Algorithms for Composite Bilevel Optimization

This paper investigates iterative methods for solving bi-level optimization problems where both inner and outer functions have a composite structure. We establish novel theoretical results, including the first analysis that provides simultaneous convergence rates for the Iteratively REgularized Proximal Gradient (IRE-PG) method, a variant of Solodov's algorithm. These rates for the inner and outer functions highlight the inherent trade-offs between their respective convergence behaviors. We further extend this analysis to an accelerated version of IRE-PG, proving faster convergence rates under specific settings. Additionally, we propose a new scheme for handling cases where these methods cannot be directly applied to the bi-level problem due to the difficulty of computing the associated proximal operator. This scheme offers surrogate functions to approximate the original problem and a framework to translate convergence rates between the surrogate and original functions. Our results show that the accelerated method's advantage diminishes under this translation.