Extensible portal frame bridge synthetic dataset for structural semantic segmentation

Abstract A number of bridges have collapsed around the world over the past years, with detrimental consequences on safety and traffic. To a large extend, such failures can be prevented by regular bridge inspections and maintenance, tasks that fall in the general category of structural health monitoring (SHM). Those procedures are time and labor consuming, which partly accounts for their neglect. Computer vision and artificial intelligence (AI) methods have the potential to ease this burden, by fully or partially automating bridge monitoring. A critical step in this automation is the identification of a bridge’s structural components. In this work, we propose an extensible synthetic dataset for structural component semantic segmentation of portal frame bridges ( PFBridge ). We first create a 3 dimensional (3D) generic mesh representing the bridge geometry, while respecting a set of rules. The definition of new, or the extension of the existing rules can adjust the dataset to specific needs. We then add textures and other realistic elements to the model, and create an automatically annotated synthetic dataset. The synthetic dataset is used in order to train a deep semantic segmentation model to identify bridge components on bridge images. The amount of available real images is not sufficient to entirely train such a model, but is used to refined the model trained on the synthetic data. We evaluate the contribution of the dataset to semantic segmentation by training several segmentation models on almost 2,000 synthetic images and then finetuning with 88 real images. The results show an increase of 28% on the F1-score when the synthetic dataset is used. To demonstrate a potential use case, the model is integrated in a 3D point cloud capturing system, producing an annotated point cloud where each point is associated with a semantic category (structural component). Such a point cloud can then be used in order to facilitate the generation of a bridge’s digital twin.