Periormer: Periodic Transformer for Seasonal and Irregularly Sampled Time Series

Time series prediction presents a significant challenge across various domains, such as transportation systems, environmental science, and multiple industrial sectors. Real-world time series data commonly exhibit periodic patterns and irregular sampling rates. Recent advancements in long sequence time series forecasting have made significant progress in adopting deep neural networks, particularly the Transformers, renowned for their robust representational capabilities. However, current Transformer-based models consider time steps as discrete tokens, thereby failing to account for periodicity and temporal intervals when selecting relevant time steps in the past. To address this limitation, we propose an end-to-end framework called Periormer for forecasting irregularly sampled time series. Periormer comprises three key components: (1) a novel input embedding layer that encodes the periodicity and time interval information, analogous to positional encoding in Transformers; (2) a feature-wise periodic attention mechanism that selects essential data points considering the periods and amplitudes of the periodic signals; and (3) a cross-feature periodic attention mechanism that identifies essential features relevant to the prediction. Experiments on four real-world datasets and one synthetic dataset demonstrate that Periormer reduces the mean squared error by 14.9% compared to state-of-the-art models.