A VLM Framework for Crack Segmentation and Description with Hierarchical Switching Mechanism

The precise monitoring of cracks is of great significance for the maintenance of infrastructure safety. However, the current crack detection methods and monitoring approaches mainly focus on the contour segmentation of cracks, lacking specific descriptions of the condition of the cracks. To tackle this problem, we propose a framework based on a vision-language model (VLM) with hierarchical switching mechanism (HSM), which has both contour segmentation and semantic description tasks. To improve the monitoring accuracy of crack analysis and optimize the fine-tuning process of the model, we also introduce a contrastive loss function involving both image and text inputs. The verification experiments conduct on the real crack dataset show that this architecture has higher accuracy and reliability for the contour segmentation of cracks compared with the traditional methods.