Multimodal Transfer Learning for Crisis Response: A Unified DeBERTa–ResNet Framework on CrisisMMD

Social media platforms generate large volumes of multimodal content during natural disasters that can support situational awareness and humanitarian decision-making. However, effectively analyzing both textual and visual information remains challenging due to weak cross-modal alignment, severe class imbalance, noisy user-generated data, and limited labeled samples. This paper presents a unified multimodal transfer-learning framework that combines a frozen DeBERTa-v3-base text encoder with a frozen ResNet-50 image encoder through a simple late-fusion strategy for crisis-related classification. The architecture projects the unimodal features into a common 256-dimensional space, concatenates them, and feeds the resulting representation into a lightweight multi-layer perceptron classifier. The framework is evaluated on all three official tasks of the CrisisMMD v2.0 dataset. Experimental results show an accuracy of 0.76 (macro-F1 = 0.74) on the informativeness task, 0.43 (macro-F1 = 0.39) on the humanitarian categories task, and 0.69 (macro-F1 = 0.64) on the damage severity task. The model performs relatively well on classes with distinctive multimodal signatures, while residual confusion remains on semantically overlapping or visually subtle categories. For the damage severity task, which is primarily image-based, the multimodal model achieves essentially the same performance as an image-only configuration, indicating limited contribution from text. The study also acknowledges that formal text-only and image-only ablation experiments under the same protocol were not conducted, so the precise benefit of fusion cannot be rigorously quantified. Overall, the work demonstrates that a computationally efficient frozen late-fusion architecture can serve as a practical and reproducible reference point for multimodal crisis informatics under the resource constraints typical of real-world disaster response.

Authors

Publication Details

Journal
International Journal of Emerging Engineering and Technology
Published
2026-11-13
DOI
https://doi.org/10.57041/8fpqd649
Primary Topic
Multimodal Machine Learning Applications
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

Multimodal Transfer Learning for Crisis Response: A Unified DeBERTa–ResNet Framework on CrisisMMD

Syed Qamar Abbas Kazmi, Haffiz_Ud_Din, Musharraf Ali, Anzal Hussain et al.
5 links
International Journal of Emerging Engineering and Technology
Multimodal Machine Learning Applications
article

Multimodal Transfer Learning for Crisis Response: A Unified DeBERTa–ResNet Framework on CrisisMMD

Syed Qamar Abbas Kazmi, Haffiz_Ud_Din, Musharraf Ali, Anzal Hussain, Sohail Ahmed, Akmal Hussain, Hannan Hussain, Syed Hassan Abbas
article en

Abstract

Social media platforms generate large volumes of multimodal content during natural disasters that can support situational awareness and humanitarian decision-making. However, effectively analyzing both textual and visual information remains challenging due to weak cross-modal alignment, severe class imbalance, noisy user-generated data, and limited labeled samples. This paper presents a unified multimodal transfer-learning framework that combines a frozen DeBERTa-v3-base text encoder with a frozen ResNet-50 image encoder through a simple late-fusion strategy for crisis-related classification. The architecture projects the unimodal features into a common 256-dimensional space, concatenates them, and feeds the resulting representation into a lightweight multi-layer perceptron classifier. The framework is evaluated on all three official tasks of the CrisisMMD v2.0 dataset. Experimental results show an accuracy of 0.76 (macro-F1 = 0.74) on the informativeness task, 0.43 (macro-F1 = 0.39) on the humanitarian categories task, and 0.69 (macro-F1 = 0.64) on the damage severity task. The model performs relatively well on classes with distinctive multimodal signatures, while residual confusion remains on semantically overlapping or visually subtle categories. For the damage severity task, which is primarily image-based, the multimodal model achieves essentially the same performance as an image-only configuration, indicating limited contribution from text. The study also acknowledges that formal text-only and image-only ablation experiments under the same protocol were not conducted, so the precise benefit of fusion cannot be rigorously quantified. Overall, the work demonstrates that a computationally efficient frozen late-fusion architecture can serve as a practical and reproducible reference point for multimodal crisis informatics under the resource constraints typical of real-world disaster response.

International Journal of Emerging Engineering and TechnologyVol. 5(1-1)
Multimodal Machine Learning Applications
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon