Resurrecting Reservoir Computing in the Deep Learning Era

Recurrent Neural Networks (RNNs) are popular for sequence modeling, but face seversal challenges, such as vanishing gradients and limited scalability due to their intrinsic sequential nature. Reservoir Computing (RC), especially Echo State Networks (ESNs), offers efficient training but has struggled to match the performance of fully trained deep learning models, also due to the difficulty to scale up the reservoir. Recent advances in State Space Models (SSMs) and Linear Recurrent Units (LRUs) have proposed parallelizable recurrent architectures. In this thesis, we introduce ESN 2.0, a complex-diagonal linear reservoir model that bridges State Space Models and Reservoir Computing. By leveraging linear recurrence, ESN 2.0 enables parallel computation across time, significantly improving efficiency while preserving expressivity. We provide theoretical insights into the model's representational power and conduct extensive empirical benchmarking, demonstrating that ESN 2.0 achieves competitive accuracy and efficiency on sequence modeling tasks compared to traditional RC models and deep learning-based recurrent models.