Asymptotic Smoothing of the Lipschitz Loss Landscape in Overparameterized One-Hidden-Layer ReLU Networks

This work studies how overparameterization reshapes the loss landscape of one-hidden-layer ReLU networks. On the theory side, it proves that for convex \(L\)-Lipschitz losses with \(\ell_1\)-regularization on the output layer, any two models at the same loss level can be connected by a continuous path with arbitrarily small excess loss \(\epsilon\), extending earlier quadratic-loss connectivity results to a broader class of objectives (including logistic/cross-entropy settings). It also derives an asymptotic bound on the energy barrier, \(\epsilon = O(m^{-\zeta})\), showing that the barrier vanishes as width \(m\) increases, so sublevel sets become connected in the infinite-width limit. Empirically, Dynamic String Sampling experiments on synthetic Moons data and the Wisconsin Breast Cancer dataset show smaller pairwise barriers for wider networks; a permutation test on the maximum gap gives \(p_{\mathrm{perm}}=0\), indicating a clear reduction in worst-case barrier height with width.