Multimodal Spatiotemporal Forecasting with Diffusion Transformer and Feature Distillation

Spatiotemporal forecasting is crucial for many real-world applications, such as predicting wildfire spread or dynamic environmental changes. Traditional methods often struggle with complex, high-dimensional data and require substantial computational resources. This paper addresses these challenges by proposing a novel three-stage framework. In the first stage, we build a multimodal dataset by converting visual data into natural language descriptions, providing richer semantic information. In the second stage, we leverage advanced image processing models to extract meaningful features from spatiotemporal sequences and apply a Diffusion Transformer for effective feature fusion. The decoder then predicts future sequences with high accuracy. In the final stage, we distill these complex features into lightweight models like UNet, ensuring computational efficiency without sacrificing prediction quality. Our approach significantly improves spatiotemporal forecasting performance while reducing computational costs, offering a promising solution for scalable and efficient real-world applications.