Fusion-Based Damage Segmentation for Multimodal Building Façade Images from an End-to-End Perspective

Multimodal image data have found widespread applications in visual-based building façade damage detection in recent years, offering comprehensive inspection of façade surfaces with the assistance of drones and infrared thermography. However, the comprehensive integration of such complementary data has been hindered by low levels of automation due to the absence of properly developed methods, resulting in high cost and low efficiency. Thus, this paper proposes an automatic end-to-end building façade damage detection method by integrating multimodal image registration, infrared–visible image fusion (IVIF), and damage segmentation. An infrared and visible image dataset consisting of 1761 pairs encompassing 4 main types of façade damage has been constructed for processing and training. A novel infrared–visible image registration method using main orientation assignment for feature point extraction is developed, reaching a high RMSE of 14.35 to align the multimodal images. Then, a deep learning-based infrared–visible image fusion (IVIF) network is trained to preserve damage characteristics between the modalities. For damage detection, a relatively high mean average precision (mAP) result of 85.4% is achieved by comparing four instance segmentation models, affirming the effective utilization of IVIF results.