Simultaneous Norm Optimality for Quadratic Regularization under Gain and Conditioning Constraints

Abstract Quadratic regularization stabilizes least-squares problems, but bounds on conditioning and coefficient amplification also constrain the attainable fit. For a fixed nonzero real design matrix, we construct an explicit penalty that minimizes every unitarily invariant norm of the fitted-target distortion under a positive penalty floor, a Hessian condition-number ceiling, and a target-to-coefficient amplification ceiling. The optimization is over all admissible symmetric penalties, including noncommuting ones. A sharp inverse-Hessian diagonal bound, attained simultaneously in all singular directions, and weak majorization establish the common optimum. We characterize the fitted-target and coefficient maps of all Frobenius minimizers and obtain uniqueness of the penalty at full column rank. For an individual target, a noncommuting penalty can improve upon this norm-optimal rule. We prove an exact convex reduction of that problem to the positive singular subspace, with a lift preserving the objective and all constraints. The explicit solution is evaluated by a singular value decomposition and scalar maxima; its relation to modified Tikhonov flooring is made precise. Independent conic computations, prescribed-spectrum checks, and comparisons with ridge and modified Tikhonov regularization examine the formulas and their numerical implications. Six fixed-feature learning benchmarks illustrate reduced training-fit distortion, while validation-selected predictive outcomes remain mixed.