On the existence of infinitely many realization functions of non-global local minima for artificial neural networks with ReLU activation

Gradient descent (GD) type optimization schemes are standard tools for training fully connected feedforward artificial neural networks (ANNs) with rectified linear unit (ReLU) activation and can be considered as temporal discretizations of solutions of gradient flow (GF) differential equations. It is known that, for shallow ReLU ANNs, the risk of every bounded GF trajectory converges under suitable assumptions to the risk of a critical point, a zero of the corresponding gradient function. This motivates studying necessary and sufficient conditions in GF and GD convergence analysis for critical points and their dependence on problem parameters, such as the target function.In the first main result, we prove for shallow ReLU ANNs that for every dimension $ d \in \mathbb{N} $, every $