Estimating urban flooding depth by integrating multimodal image-text data: A segment-level direct preference optimization-based multimodal large language model

Urban flood mapping Massive and multi-dimensional social media data provide precious opportunities for the rapid collection and assessment of urban flooding depth. However, effectively and robustly estimating water depth from these multimodal data remains a significant challenge. Although previous studies integrated several existing models, they increase model complexity and hinder joint optimization across different modalities. This paper proposes a Segment-level Direct Preference Optimization-based Multimodal Large Language Model (SDPO-MLLM) for estimating flood depth by integrating image-text data. Our contributions include the design of a hybrid training strategy combining Supervised Fine-Tuning (SFT) and SDPO to reduce inaccurate responses. Additionally, a novel structured workflow is designed, including: (1) dataset preprocessing and construction; (2) event-based extraction of flood location and depth descriptions from text; (3) generation of water depth descriptions from images and videos; (4) classification of water depth descriptions based on multiple reference objects; and (5) quantification of depth categories into numerical values. Empirical experiments are conducted on a dataset containing 2843 text records and 1563 images. The evaluation results show that SDPO-MLLM outperforms other unimodal methods, generating structured and organized results from text, and identifying flooding depth from images based on reference objects. As a case study in Wuhan, Shenzhen and Beijing, the multimodal water depth extracted from social media data is quantified and fused to map and analyze waterlogging-prone areas, demonstrating satisfactory generalization and adaptability of the developed model under various flood scenarios. Our research offers valuable insights for rapid mapping and analysis of urban waterlogging severity.