Delayformer: Spatiotemporal Transformation for Predicting High‐Dimensional Dynamics

Predicting time series is of great importance in various scientific and engineering fields. However, in the context of limited and noisy data, accurately predicting the dynamics of all variables in a high-dimensional system is a challenging task due to their nonlinearity and complex interactions. This study introduces the Delayformer framework for simultaneously predicting the dynamics of all variables by developing a novel multivariate spatiotemporal information (mvSTI) transformation that makes each observed variable into a delay-embedded state (vector) and further cross-learns those states from different variables. From a dynamical systems viewpoint, Delayformer predicts system states rather than individual variables, thus theoretically and computationally overcoming such nonlinearity and cross-interaction problems. Specifically, it first utilizes a single shared Visual Transformer (ViT) encoder to cross-represent dynamical states from observed variables in a delay-embedded form and then employs distinct linear decoders for predicting next states, i.e., equivalently predicting all original variables in parallel. By leveraging the theoretical foundations of delay embedding theory and the representational capabilities of Transformers, Delayformer outperforms current state-of-the-art methods in forecasting tasks on both synthetic and real-world datasets. Furthermore, the potential of Delayformer as a foundational time-series model is demonstrated through cross-domain forecasting tasks, highlighting its broad applicability across various scenarios.