Bottom-up building exposure modeling with multimodal earth vision

Effective disaster mitigation and management rely on up-to-date exposure models providing detailed and spatially localized information on vulnerability-relevant characteristics of buildings. This study investigates the potential of heterogeneous multimodal geo-image data—incorporating street-level imagery (SLI), very high-resolution optical remote sensing data, and a normalized digital surface model—for generic large-area building characterization. We introduce a deep multimodal multitask learning methodology for the synergistic fusion of multi-sensor data and efficient multi-criteria building classification. The proposed task-wise modality attention (TMA) fusion optimizes multimodal feature representations for individual inference tasks separately according to their specific requirements. To address the challenge of partially missing SLI data (i.e., the missing modality problem), a transformer-based SLI spatial context encoder leverages spatial correlations between structural building attributes and their visual manifestations to make the semantic information from available SLI widely accessible. With the earthquake-prone metropolis Santiago de Chile as test site, the two scenarios—SLI available and SLI missing—are evaluated through a comprehensive experimental cross-comparison of estimated generalization accuracies for classifying buildings according to five target variables: height, lateral load-resisting system material, seismic building structural type, roof shape, and block position. The results underscore the significant potential of the employed modalities and methods. Across the five addressed attributes, covering a total of 35 thematic classes, the most accurate models achieve mean κ accuracies of 85.19% and 74.96% for data points with and without SLI coverage, respectively. The presented data and methods allow to generate an area-wide building exposure model with a unique combination of thematic resolution, spatial detail and coverage.