CReformer: Convolutional Recurrent Embedding Transformer for Multivariate Long-Term Time Series Forecasting

Although transformer-based methods have made significant progress in multivariate time series forecasting, the reliance of Transformer on positional embeddings limits its ability to represent time series, especially when the look-back window is long. To address that, we propose a novel method termed as Convolutional Recurrent Embedding Transformer (CReformer). Based on the vanilla Transformer, we propose two innovations. First, CNN shows strong capabilities in capturing short-term patterns, and RNN leverages its memory mechanisms to model long-period patterns. Combining both strengths, we propose the Convolutional Recurrent Embedding (CRE) block. The proposed CRE method takes both local and global features into account, enhancing the model's predictive capability. Moreover, the model is more robust to noise interference and can adapt to multi-scale pattern features. Second, empirically, we demonstrate that masking the diagonal elements of the attention score matrix benefits time series forecasting. We refer to this attention mechanism as Diagonal Self-Attention (DA). By conducting experiments on challenging real-world datasets, our approach surpasses the baseline forecaster models in multivariate time series forecasting. Furthermore, we notice that the performance of CReformer enhances with longer sequence lengths, underscoring its scalability and capability in managing long-term dependencies.