Neural networks for singular perturbations

Abstract We prove deep neural network (DNN for short) expressivity rate bounds for solution sets of a model class of singularly perturbed, elliptic two-point boundary value problems, in Sobolev norms, on the bounded interval $$(-1,1)$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:mrow> <mml:mo>(</mml:mo> <mml:mo>-</mml:mo> <mml:mn>1</mml:mn> <mml:mo>,</mml:mo> <mml:mn>1</mml:mn> <mml:mo>)</mml:mo> </mml:mrow> </mml:math> . We assume that the given source term and reaction coefficient are analytic in $$[-1,1]$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:mrow> <mml:mo>[</mml:mo> <mml:mo>-</mml:mo> <mml:mn>1</mml:mn> <mml:mo>,</mml:mo> <mml:mn>1</mml:mn> <mml:mo>]</mml:mo> </mml:mrow> </mml:math> . The expression rate bounds in Sobolev norms in terms of the NN size are robust, i.e. uniform with respect to the singular perturbation parameter $$\varepsilon \in (0,1]$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:mrow> <mml:mi>ε</mml:mi> <mml:mo>∈</mml:mo> <mml:mo>(</mml:mo> <mml:mn>0</mml:mn> <mml:mo>,</mml:mo> <mml:mn>1</mml:mn> <mml:mo>]</mml:mo> </mml:mrow> </mml:math> for several classes of DNN architectures. In particular, ReLU NNs, spiking NNs, and $$\tanh $$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:mo>tanh</mml:mo> </mml:math> - and sigmoid-activated NNs. The latter activations can represent “exponential boundary layer solution features” explicitly, in the last hidden layer of the DNN, i.e. in a shallow subnetwork, and afford improved robust expression rate bounds in terms of the NN size. All DNN architectures allow robust exponential solution expression in so-called ‘energy’ as well as in ‘balanced’ Sobolev norms, for analytic input data.