Multimodal situation awareness for big data environments via adaptive sampling and dual self-distillation

Situation awareness (SA) in disaster management requires robust modeling of multimodal big data from complex environments. With the increasing availability of large-scale video and audio data from surveillance systems, drones, and public sensors, multimodal approaches have become essential for improving reliability. However, existing methods often rely on uniform data sampling and treat modalities equally, limiting their effectiveness when data quality varies across time and modality. Compared with our previous work, this study extends from modality-specific analysis to a multimodal audio–visual framework. To address these challenges, this paper proposes MSA-SD², a multimodal situation awareness framework specifically designed for big data environments based on selective dual self-distillation. An adaptive temporal sampling and alignment module is introduced to prioritize informative temporal segments and ensure consistent audio–visual correspondence, enabling efficient processing of large-scale multimodal big data. For visual representation, a superpixels-guided vision transformer is employed, while a transformer-based audio encoder models temporal acoustic patterns. A multimodal student model is guided by visual and audio teacher models through a selective distillation strategy that adapts to modality reliability. Experimental results demonstrate improved performance on large-scale multimodal datasets, particularly under challenging big data conditions with varying data quality across modalities.

Authors

Institutions

Publication Details

Journal
Journal Of Big Data
Published
2026-11-16
DOI
https://doi.org/10.1186/s40537-026-01607-1
Primary Topic
Multimodal Machine Learning Applications
Type
article
Field-Weighted Citation Impact
0.00

Funders

Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

Multimodal situation awareness for big data environments via adaptive sampling and dual self-distillation

Jieli Chen, Chee Shen Lim, Kah Phooi Seng, Li-Minn Ang et al.
2 links
Journal Of Big Data
Multimodal Machine Learning Applications
article

Multimodal situation awareness for big data environments via adaptive sampling and dual self-distillation

Jieli Chen, Chee Shen Lim, Kah Phooi Seng, Li-Minn Ang, Jeremy Smith
article en

Abstract

Situation awareness (SA) in disaster management requires robust modeling of multimodal big data from complex environments. With the increasing availability of large-scale video and audio data from surveillance systems, drones, and public sensors, multimodal approaches have become essential for improving reliability. However, existing methods often rely on uniform data sampling and treat modalities equally, limiting their effectiveness when data quality varies across time and modality. Compared with our previous work, this study extends from modality-specific analysis to a multimodal audio–visual framework. To address these challenges, this paper proposes MSA-SD², a multimodal situation awareness framework specifically designed for big data environments based on selective dual self-distillation. An adaptive temporal sampling and alignment module is introduced to prioritize informative temporal segments and ensure consistent audio–visual correspondence, enabling efficient processing of large-scale multimodal big data. For visual representation, a superpixels-guided vision transformer is employed, while a transformer-based audio encoder models temporal acoustic patterns. A multimodal student model is guided by visual and audio teacher models through a selective distillation strategy that adapts to modality reliability. Experimental results demonstrate improved performance on large-scale multimodal datasets, particularly under challenging big data conditions with varying data quality across modalities.

Journal Of Big Data
University of Liverpool (GB), University of the Sunshine Coast (AU), Taylor's University (MY), Xi’an Jiaotong-Liverpool University (CN)
University of the Sunshine Coast
Multimodal Machine Learning Applications
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon