A multimodal LLM framework with time-series-guided visual routing for ultra-short-term solar forecasting

Accurate ultra-short-term solar irradiance forecasting is important for photovoltaic power management, storage scheduling, and the secure operation of power systems with increasing renewable penetration. However, historical irradiance measurements primarily reflect atmospheric variations after they have affected the observation site, providing limited information about approaching clouds that may induce rapid irradiance ramps. Although ground-based sky images provide complementary spatial information, using complete visual representations can introduce irrelevant regions and weaken effective multimodal interaction. This study proposes a time-series-guided multimodal forecasting framework that integrates historical irradiance, sky images, statistical prompts, and a frozen LLM. Historical irradiance sequences are divided into temporal patches and reprogrammed into the LLM representation space, while a hierarchical visual encoder extracts spatial tokens from sky images. A time-series-conditioned visual router evaluates the relevance of individual image patches according to the current temporal representation and retains task-relevant regions through hard Top- K selection with a differentiable straight-through estimator. The selected visual tokens are then integrated with the temporal representation through TS-guided cross-modal attention, in which temporal tokens serve as queries and visual tokens provide keys and values. A residual connection preserves the original temporal information, and statistical prompt embeddings are subsequently concatenated with the fused temporal tokens before processing by the frozen LLM. Experiments on the Folsom dataset achieve nRMSE values of 14.89%, 17.19%, and 19.91% at 10-, 30-, and 60-min forecasting horizons, respectively, outperforming all evaluated baselines. Ablation and sensitivity analyses further confirm the contributions of time-series-conditioned visual selection and explicit cross-modal fusion, while an intermediate number of selected patches provides a favorable balance between information retention and visual redundancy. Cross-dataset validation on the Stanford SKIPP’D dataset further demonstrates the applicability of the proposed framework to PV power forecasting under a different site and prediction target.