Scaled and Graduated Learning in Deep RELU Networks and Reconstructing Depp Inelastic Scattering Kinematics

To address computational challenges in learning deep neural networks, properties of deep RELU networks were studied to develop a multi-scale learning model. The multi-scale model was compared to the multi-grade learning models. Unlike the deep neural network learned from the standard single-scale, single-grade model, the multi-scale neural networks use low scale information from all hidden layers, and thusly provide a robust approximation method that requires fewer parameters, lower computational time, and is resistant to noise. It is shown that the multiscale method is not subject to issues arising from the vanishing gradient problem. This allows very deep multi-scale networks to be effectively trained. It is proven that the collection of multi-scale neural networks are universal approximators in the space of continuous functions. The neural network learned from a multi-grade model is the superposition of the neural networks, in a stair-shape, each of which is learned from one grade of the learning. Three proof-of-concept numerical examples presented in the paper demonstrate that the multi-scale and multi-grade methods are superior to the single-scale, single-grade networks. The extended analysis on reconstructing kinematic observables in deep inelastic scattering kinematics with multi-scale neural networks shows that not only are those models effective for real world problems, but the power of the approximation is sufficiently great, that it can outperform reconstruction methods based on physical laws.