Semantic-Enriched Image Retrieval for Bridge Damage Assessment

Periodic bridge damage inspections result in a vast number of image records stored in a database, which can be used as a reference for the subsequent damage assessment. However, the currently used content-based image retrieval (CBIR) techniques are limited by the 'semantic gap'. They tend to consider only the low-level visual features in an image and ignore the high-level semantic information. This study proposes a semantic-enriched image retrieval framework (SEIR-Net) for bridge damage assessment. The framework enables the image encoder to extract low-level visual features and high-level semantic information by fine-tuning the multi-modal image captioning model (CNN-LSTM). The high-dimensional vectors extracted using the fine-tuned encoder are stored in the FAISS vector database, and efficient retrieval is achieved based on L2 Euclidean distance. Retrieval evaluation was performed on a damage dataset constructed on the real-world bridge inspection report, and our proposed method outperforms the commonly used VGG-16 and ResNet-50 models on the mAP and Recall@K (K=1, 2, 4) metrics. These results suggest that incorporating the semantic content of damage in image retrieval would be more beneficial for assessment references. In summary, this study effectively enhances the utility of historical image records in bridge damage assessment through semantic-enriched image retrieval techniques.