KNN-augmented attention transformer for hybrid deep-learning-based meteorological forecasting

Recent advances in deep learning have significantly impacted weather forecasting, where accurate predictions are crucial for effective decision-making across various sectors. This paper presents a novel approach to weather forecasting by introducing an advanced transformer-based model, which harnesses the capabilities of the Focused Transformer (FOT) framework with significant adaptations for meteorological applications. Central to our method is the substitution of the exact k-nearest-neighbor (k-NN) lookup with an approximate k-NN method, enhancing the model’s efficiency and scalability. This modification enables the transformer to dynamically integrate expansive temporal and spatial data sequences more effectively, crucial for accurate weather predictions. We adopt the contrastive training technique to refine the context scaling, which is critical for processing the complex dynamics of weather patterns. Performance evaluations on several weather datasets demonstrate that our model achieves superior forecasting accuracy, indicated by improvements in root mean squared error and mean absolute error metrics, compared to existing state-of-the-art models. Our findings suggest that applying transformer models with approximate k-NN lookups offers a promising direction for developing more robust and efficient weather forecasting systems.