WaveRoRA: Wavelet Rotary Route Attention for Multivariate Time Series Forecasting

Various sensors of Internet of Things (IoT) generate massive amounts of mobile traffic data, forming multivariate time series (MTS). Accurate forecasting of MTS facilitates the enhancement of proactive autoscaling and resource allocation in edge networks. While recent Transformer-based models (Transformers) have achieved significant success in MTS forecasting (MTSF), they tend to rely solely on either time-domain or frequency-domain features, which captures inadequate trends and periodic characteristics. To this end, we propose a wavelet learning framework that seamlessly integrates wavelet transforms with Transformers to benefit from time and frequency characteristics. We design a mixing-splitting architecture to model multi-scale wavelet coefficients and utilizes the attention mechanism to capture inter-series dependencies in the wavelet domain. However, the vanilla softmax self-attention (SA) is high-computational-cost and its smoothing effect diminishes the contrast between strong and weak variable correlations. Therefore, we propose a novel attention mechanism: Rotary Route Attention (RoRA). RoRA incorporates rotary positional embeddings to enhance feature diversity and introduces a small number of routing tokens <inline-formula><tex-math notation="LaTeX">$r$</tex-math></inline-formula> to aggregate information from the <inline-formula><tex-math notation="LaTeX">$KV$</tex-math></inline-formula> matrices and redistribute it to the <inline-formula><tex-math notation="LaTeX">$Q$</tex-math></inline-formula> matrix. Such design strengthens interactions among strongly correlated variables while mitigating the impact of weakly correlated noise. We further propose WaveRoRA, a unified model that leverages RoRA capturing inter-series dependencies in the wavelet domain. We conduct extensive experiments on eight real-world datasets. The results indicate that WaveRoRA outperforms existing state-of-the-art models while maintaining lower computational costs.