CQEformer: A Causal and Query-Enhanced Transformer Variant for Time-Series Forecasting

Structural breaks and volatility clustering are fundamental challenges in time series analysis. We propose CQEformer, an encoder-only Transformer variant for time-series modeling that addresses these challenges via two complementary innovations. First, the causal residual embedding (CRE) ensures temporal causality and improves local adaptivity to abrupt structural changes. Second, the query-enhanced multi-head self-attention (QEAttention) incorporates multi-order moment statistics and entropy to guide attention dynamically toward high-volatility regions while preserving global dependence structures. For parameter optimization, we derive analytical gradients for all components and update them using the Adam stochastic optimization algorithm. Empirical evaluations on financial time series datasets and the public Traffic dataset show that CQEformer consistently outperforms established baselines, including LSTM, GRU, TCN, and the standard Transformer. Time-window sensitivity analyses demonstrate the robustness of the framework, while ablation studies further confirm that the proposed modules are complementary and contribute to improved forecasting performance across different volatility regimes.