Insight into semi-convergence of iterative regularization methods

We present a tutorial analysis that gives insight into the behavior of semi-convergence for iterative regularization methods, where the number of iterations acts as a regularization parameter. It is generally observed that when we apply an iterative solver to a linear inverse problem with noisy data, then initially the iteration vectors approach the noise-free solution – while at later stages the iterates converge to a very noisy solution. By stopping the regularizing iterations toward the end of the first phase, we can obtain a good approximation to the solution. An important ingredient is a simple statistical analysis that gives insight into why, when, and how semi-convergence manifests itself. The main focus is on the CGLS algorithm for solving linear least squares problems.