Predicting complex time series with deep echo state networks

Although many real-world time series are complex, developing methods that can learn from their behavior effectively enough to enable reliable forecasting remains challenging. Recently, several machine-learning approaches have shown promise in addressing this problem. In particular, the echo state network (ESN) architecture, a type of recurrent neural network where neurons are randomly connected and only the read-out layer is trained, has been proposed as suitable for many-step-ahead forecasting tasks. ESNs have been shown to be effective in predicting complex time series while reducing training costs, thereby improving efficiency. Deep echo state networks, which extend the architecture of ESNs by stacking multiple reservoir layers, have also been proposed as an option to learn different features of the dynamics across the layers, but to date they have received less study. In this work, we analyzed the performance of deep ESNs for several variations in network structure, including a hybrid approach that integrated a knowledge-based model. We found that for specific network structures, deep ESNs could improve prediction accuracy over baseline ESNs with the same number of neurons by up to 65% for a chaotic data set derived from the Mackey-Glass model and 14% for an experimental data set of complex zebrafish cardiac voltage recordings. Similarly, deep hybrid ESNs could reduce error compared to same-size flat hybrid ESNs by up to 59% for the Mackey-Glass data set and 11% for the experimental data set. Our results also showed that the hybrid approach was especially beneficial for the experimental data set and that deep network structures improved prediction robustness.