CracksGPT: Exploring the Potential and Limitations of Multimodal AI for Building Crack Analysis

Building cracks are among the critical building defects, as they can compromise structural integrity, occupant safety and building sustainability. Traditional laborious building inspection methods are cumbersome and erroneous. Computer vision-based crack detection relies on image recognition and does not analyse the underlying causes or suggest rectification strategies. This study explores the potential and limitations of multimodal AI models, that integrate image and text modalities for building crack analysis. As a proof-of-concept, the vision–language model, CracksGPT was built upon a fine-tuned MiniGPT-v2. It was trained on custom crack images with text descriptions detailing visual features, possible causes, and rectification options. It was tested on crack images from a building site in Sydney. When provided with an image of a wall crack, CracksGPT can classify crack patterns of vertical, horizontal, diagonal, and stair-step and interpret possible underlying causes with potential rectification strategies. The ROUGE metric was used for language generation quality evaluation followed by a performance evaluation by building inspection experts. The model’s performance is sensitive to input image quality and training data limitations, specifically in complex scenarios, reaffirming the value of expert overseeing. The findings highlight the potential and limitations of multimodal AI for integrating vision–language reasoning into building inspections.