Constraint Unlearnability in Overparameterized Neural Networks

We study whether overparameterized neural networks can learn to satisfy pointwise constraints without inductive bias. One might expect that a sufficiently expressive network would learn to respect such constraints. We prove that this is not the case. For overparameterized feedforward neural networks with real-analytic activation functions, assuming the manifold of global minima is connected, the set of parameters for which the network satisfies a real-analytic pointwise constraint has measure zero on this manifold. Since gradient descent converges to this manifold, it almost surely finds solutions that violate the constraint. The proof applies the identity theorem for real-analytic functions twice: first, to show that a network either satisfies the constraint everywhere or violates it almost everywhere; second, to show that overparameterization excludes the everywhere-satisfied case. We call this phenomenon constraint unlearnability—despite interpolating the training data, the network cannot satisfy the constraint exactly on the entire input domain.