Quantifying expressive power of time-series neural networks

To quantify and measure the expressive power of time-series neural networks that built on recurrent structure, feed-forward structure, and self-attention mechanism, this work proposes three indices of trajectory length, squared normalized length and angular entropy. With the indices, some conclusions are revealed as: (i) The ability of a model to maintain the angular distance between the scaled features determines the time-series feature extraction, and further determines the forecasting performance. (ii) The angular entropy has logarithmic correlation to the sequence length of the features. Its value approximates log 𝜋 if the sequence length increases. (iii) The neural networks with feed-forward structure are sufficient for the forecasting task, because the models have relatively lower angular entropy. The code of this work is available at: https://github.com/KMY-SEU/EXP-CAIMLR2024.