Semantic Part Segmentation of Heritage Monuments from Crowdsourced Images: Consistent Labeling using SfM-guided SAM

Modeling of heritage monuments in 3D poses a computationally complex task. One approach to reduce complexity lies in modeling architecturally significant parts and integrating those into a whole. To complete that task based on crowdsourced images without annotations and with varying viewpoints and lighting, we consider semantic part segmentation of heritage monuments using the state-of-theart Segment Anything Model (SAM). However, SAM does not produce semantically consistent labels across multiple images. As a remedy, we propose an SfM-guided framework that combines sparse 3D reconstruction from Structure-from-Motion (SfM) with SAM to achieve consistent part-level segmentation of architectural elements. Using 3D spatial context and camera poses, our method aligns segmentations across diverse images, bridging the gap between fine-grained segmentation and semantic consistency. This scalable approach demonstrates the potential to integrate 3D geometry with foundational models for cultural heritage analysis.