Compute KV First: Scaling up Time Series Model with Cross-Retention-only Transformer

Transformers have significantly advanced time series forecasting, but still face challenges such as quadratic time complexity, high memory consumption, and lack of positional priors. While recent research has explored Cross-Attention-only architectures that outperform complex Transformer models, the quadratic complexity of vanilla attention remains a bottleneck for scaling time series Transformers. With the rapid development of large language models, research in the NLP field has focused on addressing the quadratic complexity issue of attention mechanisms. Inspired by the recent Retentive Network (RetNet) in the field of NLP, we propose CKVF, a Cross-Retention-only Time Series Transformer that achieves linear complexity by computing key-value (KV) representations first, yet retains global attention capability. Building on RetNet’s retention mechanism, we introduce Cross-Retention, an adaptation of RetNet’s retention mechanism. Cross-Retention avoiding the high complexity introduced by attention matrices. Furthermore, we incorporate a self-adaptive adjacency matrix within Cross-Retention to capture dependencies between variables more effectively. Extensive experiments thoroughly validate that CKVF achieves state-of-the-art performance across various benchmark datasets.