Balancing Adaptivity and Efficiency: Revisiting Sparsity in Memory-Augmented Transformers for Time Series Forecasting

Transformer-based models have shown remarkable success in time series forecasting, particularly through recent innovations that integrate memory mechanisms and learnable sparse attention. LTSMiTransformer exemplifies this trend by combining a memory module with a learnable temporal sparsity mechanism to enhance long-sequence modeling. In this study, we introduce TSMiTransformer, an ablation variant of LTSMiTransformer that retains the memory module and sparsity structure but replaces the learnable sparsity with a fixed sparsity factor—determining the number of attention links proportionally to the input length. This design choice offers a more interpretable and computationally efficient alternative, enabling a focused investigation into the value of sparsity learnability. Across eight benchmark datasets, TSMiTransformer demonstrates strong and consistent performance, often remaining competitive with its more complex counterpart. These findings underscore the robustness of memory-augmented attention and suggest that fixed temporal sparsity can retain much of the forecasting capability, even without full adaptability. Codes available on GitHub.