Deep Residual Echo State Networks: exploring residual orthogonal connections in untrained RNNs

The Reservoir Computing (RC) paradigm offers a unique approach for the design of randomized Recurrent Neural Networks (RNN). In particular, Echo State Networks (ESN), a type of untrained RNNs within the RC paradigm, have become increasingly popular over the last decade due to their lightweight, fast and efficient learning, where only a simple linear readout layer requires optimization. This thesis introduces a novel class of deep residual RC models, called Deep Residual Echo State Networks (DeepResESN). DeepResESN integrates residual orthogonal connections within layered ESNs, creating a hierarchy of residual untrained recurrent layers. This work contributions can be summarized as follows. (1) We introduce DeepResESN, a novel class of deep residual ESNs. (2) We propose a theoretical analysis that generalizes the Echo State Property (ESP) to the proposed model, outlining necessary and sufficient conditions for ensuring stable and contractive dynamics within DeepResESN. Then, (3) we investigate the architectural bias introduced by deeper layers and by different types of orthogonal matrices, leveraging eigenspectrum analysis and spectral frequency analysis tools. Finally, (4) we validate the proposed approach through a series of heterogeneous tasks on time series, including memory based, forecasting and classification. Experimental results demonstrate the advantageous architectural bias introduced by DeepResESN, achieving consistent performance improvements over other RC models.