Sparse-to-dense cross-attention for multimodal climate representation learning and data fusion
Integrating multimodal environmental observations poses severe data engineering challenges, including sparse policy document context, varying social text streams, and inconsistent temporal records. In this work, we present an end-to-end multimodal data fusion framework that uses exponential-decay weighting (α = 0.7) for historical text imputation, along with a sparse-to-dense cross-attention mechanism, across 12,684 policy document chunks. Evaluated on national-level climate data from Ukraine, our results show that incorporating cross-attention provides crucial contextual regularization and improves numerical representation robustness, reducing test prediction error by approximately half (MSE = 0.1477 ± 0.0188) compared to the non-attention transformer-based baseline (MSE = 0.2910 ± 0.045). However, for the self-supervised data fusion approach evaluated on reconstruction loss, the no cross-attention approach yields the lowest error (MSE = 0.0784 ± 0.0016) compared with other modality sets (MSE ≈ 0.2938 ± 0.0397) due to the specifics of self-supervised learning evaluation. Furthermore, sensitivity analysis demonstrates that expanding the retrieval window [ K ε (200, 1,000)] yields higher attention entropy, preventing attention collapse across multi-modal environmental applications.
Authors
- Taras Ustyianovych (ORCID: https://orcid.org/0000-0002-6323-7924)
- Solomiia Fedushko
Institutions
- Comenius University Bratislava (SK)
- Lviv Polytechnic National University (UA)
Publication Details
- Journal
- Frontiers in Big Data
- Published
- 2026-11-20
- DOI
- https://doi.org/10.3389/fdata.2026.1968082
- Primary Topic
- Multimodal Machine Learning Applications
- Type
- article
- Field-Weighted Citation Impact
- 0.00