Exponential expressivity of ${\rm ReLU}^k$ neural networks on Gevrey classes with point singularities

We analyze deep Neural Network emulation rates of smooth functions with point singularities in bounded, polytopal domains \(\mathrm{D} \subset \mathbb{R}^d\), \(d=2,3\). We prove exponential emulation rates in Sobolev spaces in terms of the number of neurons and in terms of the number of nonzero coefficients for Gevrey-regular solution classes defined in terms of weighted Sobolev scales in \(\mathrm{D}\), comprising the countably-normed spaces of I.M. Babu\v{s}ka and B.Q. Guo. As intermediate result, we prove that continuous, piecewise polynomial high order (``\(p\)-version'') finite elements with elementwise polynomial degree \(p\in\mathbb{N}\) on arbitrary, regular, simplicial partitions of polyhedral domains \(\mathrm{D} \subset \mathbb{R}^d\), \(d\geq 2\) can be \emph{exactly emulated} by neural networks combining ReLU and ReLU\(^2\) activations. On shape-regular, simplicial partitions of polytopal domains \(\mathrm{D}\), both the number of neurons and the number of nonzero parameters are proportional to the number of degrees of freedom of the finite element space, in particular for the \(hp\)-Finite Element Method of I.M. Babu\v{s}ka and B.Q. Guo.