Time Series Forecasting Using Recurrent Neural Networks Based on Recurrent Sigmoid Piecewise Linear Neurons

We propose a new recurrent sigmoid piecewise linear neuron that can be used in neural networks to perform time series forecasting. The neuron model guarantees its dynamical stability for any sequence of input values and any number of recurrent steps and provides an upper bound for the variance of context vector elements. The neuron can be used as a drop-in replacement for the popular long short-term memory and gated recurrent unit neurons. In addition to theoretical analysis experiments on real-world time series were performed to evaluate networks with different structures and neuron types. Experiments show that networks with the new neuron achieve better test accuracy while using a considerably smaller number of trainable parameters.