Multimodal situation awareness for big data environments via adaptive sampling and dual self-distillation
Situation awareness (SA) in disaster management requires robust modeling of multimodal big data from complex environments. With the increasing availability of large-scale video and audio data from surveillance systems, drones, and public sensors, multimodal approaches have become essential for improving reliability. However, existing methods often rely on uniform data sampling and treat modalities equally, limiting their effectiveness when data quality varies across time and modality. Compared with our previous work, this study extends from modality-specific analysis to a multimodal audio–visual framework. To address these challenges, this paper proposes MSA-SD², a multimodal situation awareness framework specifically designed for big data environments based on selective dual self-distillation. An adaptive temporal sampling and alignment module is introduced to prioritize informative temporal segments and ensure consistent audio–visual correspondence, enabling efficient processing of large-scale multimodal big data. For visual representation, a superpixels-guided vision transformer is employed, while a transformer-based audio encoder models temporal acoustic patterns. A multimodal student model is guided by visual and audio teacher models through a selective distillation strategy that adapts to modality reliability. Experimental results demonstrate improved performance on large-scale multimodal datasets, particularly under challenging big data conditions with varying data quality across modalities.
Authors
- Jieli Chen (ORCID: https://orcid.org/0009-0002-8883-4407)
- Chee Shen Lim (ORCID: https://orcid.org/0000-0003-3033-4688)
- Kah Phooi Seng (ORCID: https://orcid.org/0000-0002-8071-9044)
- Li-Minn Ang
- Jeremy Smith
Institutions
- University of Liverpool (GB)
- University of the Sunshine Coast (AU)
- Taylor's University (MY)
- Xi’an Jiaotong-Liverpool University (CN)
Publication Details
- Journal
- Journal Of Big Data
- Published
- 2026-11-16
- DOI
- https://doi.org/10.1186/s40537-026-01607-1
- Primary Topic
- Multimodal Machine Learning Applications
- Type
- article
- Field-Weighted Citation Impact
- 0.00
Funders
- University of the Sunshine Coast