3D segmentation method for stone cultural relics utilizing multi-modal feature enhancement based on SAM

Abstract To enhance segmentation in 3D reconstruction, especially for dense textures, blurred edges, and small structures, this study proposes MF-ESG, a non-invasive method for multi-view image acquisition and 3D segmentation. It introduces the SAGa framework, which combines 3DGS with SAM’s zero-shot capabilities to produce low-dimensional masks from 2D images, guiding multimodal 3D feature learning. This improves adaptability across various stone relic scenes. The edge-sensitive MLLS module, incorporating multi-scale large-kernel attention and dynamic receptive fields, enhances edge segmentation in complex environments. On SPIn-NeRF, NVOS, and custom datasets, the method achieves average mIoU scores of 94.1%, 94.3%, and 95.75%, surpassing SAGa by 0.7%, 1.7%, and 1.23%, respectively. It also significantly reduces edge defects and central voids in reconstruction.