Asymptotics of Proximity Operator for Squared Loss and Performance Prediction of Nonconvex Sparse Signal Recovery

Proximal splitting-based convex optimization is a promising approach to linear inverse problems because we can use some prior knowledge of the unknown variables explicitly. An understanding of the behavior of the optimization algorithms would be important for the tuning of the parameters and the development of new algorithms. In this paper, we first analyze the asymptotic property of the proximity operator for the squared loss function, which appears in the update equations of some proximal splitting methods for linear inverse problems. Our analysis shows that the output of the proximity operator can be characterized with a scalar random variable in the large system limit. Moreover, we apply the asymptotic result to the prediction of optimization algorithms for compressed sensing. Simulation results demonstrate that the MSE performance of the Douglas-Rachford algorithm can be well predicted in compressed sensing with the ℓ1 optimization. We also examine the behavior of the prediction for the case with nonconvex smoothly clipped absolute deviation (SCAD) and minimax concave penalty (MCP) regularization.