SAM-based Segmentation of Multi-Class Bridge Components from Diverse Real-Scene Inspection Images

Traditional bridge inspection methods rely on manual visual inspection, which is time-consuming, labor-intensive, and potentially dangerous. Automated inspection approaches, which use unmanned aerial vehicles (UAVs) and computer vision, aim to address this issue. However, three knowledge gaps remain. First, although considerable research has been conducted on defect detection and segmentation in bridge inspection, there has been limited focus on segmenting and characterizing specific bridge components that contain defects. Such segmentation provides essential contextual information for understanding the importance of defects for maintenance decision making. Second, existing bridge component recognition approaches face challenges in generalizing across various scenarios, especially in close-range inspections where contextual information is often missing. Third, current developments in the foundation models in the computer vision, such as the segment anything model (SAM), remain unexplored for bridge component segmentation from inspection images due to its lack of domain-specific knowledge and unable to assign semantic labels to multiple segmented components. To address these limitations, this paper proposes a SAM-based image segmentation method for multi-class bridge component segmentation from diverse bridge inspection images. This method leverages the SAM architecture and pre-training from Segment Anything 1 Billion (SA-1B) to enhance feature extraction and improve generalizability. The method also integrates a U-Net decoder to address the challenges of multi-class bridge component segmentation. The proposed method was trained and tested end-to-end on seven classes based on the FHWA’s Bridge Inspector’s Reference Manual. The results demonstrate promising performance, indicating the potential of this SAM-based approach for efficient and accurate bridge component segmentation.