Regularisation and the least squares problem in the analysis of echo state networks

Abstract An echo state network ( <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" overflow="scroll"> <mml:mi mathvariant="normal">ESN</mml:mi> </mml:math> ) is a recurrent neural network that has several advantages with respect to a deep neural network, including its fast training phase and the absence of vanishing and exploding gradients. The training phase reduces to solving the least squares (LS) problem <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" overflow="scroll"> <mml:msub> <mml:mrow> <mml:mi mathvariant="bold">w</mml:mi> </mml:mrow> <mml:mrow> <mml:mi>ls</mml:mi> </mml:mrow> </mml:msub> </mml:math> = <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" overflow="scroll"> <mml:mi>arg</mml:mi> <mml:msub> <mml:mrow> <mml:mi>min</mml:mi> </mml:mrow> <mml:mrow> <mml:mi mathvariant="bold">v</mml:mi> </mml:mrow> </mml:msub> <mml:msubsup> <mml:mrow> <mml:mfenced close="∥" open="∥"> <mml:mrow> <mml:mi mathvariant="bold">vX</mml:mi> <mml:mo>−</mml:mo> <mml:mi mathvariant="bold">y</mml:mi> </mml:mrow> </mml:mfenced> </mml:mrow> <mml:mrow> <mml:mn>2</mml:mn> </mml:mrow> <mml:mrow> <mml:mn>2</mml:mn> </mml:mrow> </mml:msubsup> </mml:math> , where X is the reservoir matrix, and X and y are functions of the training data. It is common to add regularisation to this problem because, it is claimed, it minimises the adverse effects of overfitting. Recent work in deep neural networks, physics informed neural networks and regression has shown, however, that regularisation does not solve the problem of overfitting, and thus this paper considers the application of regularisation to <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" overflow="scroll"> <mml:mi mathvariant="normal">ESN</mml:mi> </mml:math> s by analysing their predictive abilities on several time series, including a non-linear communication channel, the Hénon map and multiple superimposed oscillations. It is shown that the solution <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" overflow="scroll"> <mml:msub> <mml:mrow> <mml:mi mathvariant="bold">w</mml:mi> </mml:mrow> <mml:mrow> <mml:mi>ls</mml:mi> </mml:mrow> </mml:msub> </mml:math> of the LS problem is, for many problems, stable with respect to a perturbation in y for a wide range of parameter values of an <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" overflow="scroll"> <mml:mi mathvariant="normal">ESN</mml:mi> </mml:math> , and thus regularisation must not be applied to these problems. Each problem must, however, be considered because the need, or otherwise, to apply regularisation is dependent on many parameters of an <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" overflow="scroll"> <mml:mi mathvariant="normal">ESN</mml:mi> </mml:math> . Furthermore, regularisation is not benign because its use when a condition on the rate of decay of the singular values of X is not satisfied leads to a large error between the theoretically exact and regularised solutions of the LS problem.