Hermite polynomials facilitating on-line learning analysis of layered neural networks with arbitrary activation function

Following the standard statistical mechanics methods we analyze the training by online stochastic gradient descent of two-layer neural networks in a student–teacher scenario. We focus on understanding the role that different activations play, in particular mismatches between the student and the teacher, in these learning scenarios. By expanding the activation functions in the Hermite polynomial basis, we are able to effectively approximate the relevant integrals with much less computational effort than naive numerical integration. Moreover, we also extend the framework to study scenarios of concept drift and weight decay also with arbitrary activation functions. All these extensions comprise relevant advances in the field, allowing us to obtain analytical results for more realistic scenarios. • We present a statistical physics based analysis of layered neural networks with arbitrary activation functions in an on-line learning scenario. • The method employs a Hermite polynomial series representation of the activation function to derive relevant integrals involved in the ODE formulation of the on-line learning. • This enables the analysis of on-linelearning for arbitrary activation function and mismatched student-teacher scenarios. • An analysis of mismatched student-teacher scenarios including drift and weight decay is provided.