Improved Universal Approximation with Neural Networks Studied via Affine-Invariant Subspaces of \(L_2(\mathbb{R}^n)\)

We show that there are no non-trivial closed subspaces of \(L_2(\mathbb{R}^n)\) that are invariant under invertible affine transformations. We apply this result to neural networks showing that any nonzero \(L_2(\mathbb{R})\) function is an adequate activation function in a one hidden layer neural network in order to approximate every function in \(L_2(\mathbb{R})\) with any desired accuracy. This generalizes the universal approximation properties of neural networks in \(L_2(\mathbb{R})\) related to Wiener's Tauberian Theorems. Our results extend to the spaces \(L_p(\mathbb{R})\) with \(p>1\).