VLMM Model-Based Comprehensive Understanding of Post-Earthquake Images in Civil Infrastructure

Artificial intelligence is playing an increasingly significant role in interpreting postearthquake images of civil infrastructure. However, current models for analyzing such disaster-related images predominantly rely on unimodal image processing, which offers limited comprehensiveness and interpretability. To address this limitation, this paper presents a vision-language multimodal model (VLMM) named VLMM-civil based on the LLaVA-1.5 architecture, designed to integrate visual and linguistic modalities for comprehensive understanding of postearthquake civil infrastructure images. This study develops a specialized dataset framework with image-instruction matching, damage object detection, and detailed description tasks. VLMM-civil is trained with low-rank adaptation fine-tuning under multiple initial learning rates and training epochs and then evaluated with metrics of the exact matching ratio, mean intersection over union, metric for evaluation of translation with explicit ordering, recall-oriented and understudy for gisting evaluation—longest common subsequence, consensus-based image description evaluation, and semantic propositional image caption evaluation. The best VLMM-civil model was demonstrated to provide accurate image-instruction matching, damage object detection, and detailed description outputs. This study not only extends the application scope of vision-language multimodal models in civil infrastructure assessment but also establishes a methodological benchmark for future multimodal studies in civil engineering.