Sparse-to-dense cross-attention for multimodal climate representation learning and data fusion

Integrating multimodal environmental observations poses severe data engineering challenges, including sparse policy document context, varying social text streams, and inconsistent temporal records. In this work, we present an end-to-end multimodal data fusion framework that uses exponential-decay weighting (α = 0.7) for historical text imputation, along with a sparse-to-dense cross-attention mechanism, across 12,684 policy document chunks. Evaluated on national-level climate data from Ukraine, our results show that incorporating cross-attention provides crucial contextual regularization and improves numerical representation robustness, reducing test prediction error by approximately half (MSE = 0.1477 ± 0.0188) compared to the non-attention transformer-based baseline (MSE = 0.2910 ± 0.045). However, for the self-supervised data fusion approach evaluated on reconstruction loss, the no cross-attention approach yields the lowest error (MSE = 0.0784 ± 0.0016) compared with other modality sets (MSE ≈ 0.2938 ± 0.0397) due to the specifics of self-supervised learning evaluation. Furthermore, sensitivity analysis demonstrates that expanding the retrieval window [ K ε (200, 1,000)] yields higher attention entropy, preventing attention collapse across multi-modal environmental applications.

Authors

Institutions

Publication Details

Journal
Frontiers in Big Data
Published
2026-11-20
DOI
https://doi.org/10.3389/fdata.2026.1968082
Primary Topic
Multimodal Machine Learning Applications
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

Sparse-to-dense cross-attention for multimodal climate representation learning and data fusion

Taras Ustyianovych, Solomiia Fedushko
2 links
Frontiers in Big Data
Multimodal Machine Learning Applications
article

Sparse-to-dense cross-attention for multimodal climate representation learning and data fusion

Taras Ustyianovych, Solomiia Fedushko
article en

Abstract

Integrating multimodal environmental observations poses severe data engineering challenges, including sparse policy document context, varying social text streams, and inconsistent temporal records. In this work, we present an end-to-end multimodal data fusion framework that uses exponential-decay weighting (α = 0.7) for historical text imputation, along with a sparse-to-dense cross-attention mechanism, across 12,684 policy document chunks. Evaluated on national-level climate data from Ukraine, our results show that incorporating cross-attention provides crucial contextual regularization and improves numerical representation robustness, reducing test prediction error by approximately half (MSE = 0.1477 ± 0.0188) compared to the non-attention transformer-based baseline (MSE = 0.2910 ± 0.045). However, for the self-supervised data fusion approach evaluated on reconstruction loss, the no cross-attention approach yields the lowest error (MSE = 0.0784 ± 0.0016) compared with other modality sets (MSE ≈ 0.2938 ± 0.0397) due to the specifics of self-supervised learning evaluation. Furthermore, sensitivity analysis demonstrates that expanding the retrieval window [ K ε (200, 1,000)] yields higher attention entropy, preventing attention collapse across multi-modal environmental applications.

Frontiers in Big DataVol. 9
Comenius University Bratislava (SK), Lviv Polytechnic National University (UA)
Climate action
Multimodal Machine Learning Applications
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon