Image-based Deep Learning Approaches for Point Cloud Classification for Heritage BIM Modelling

Abstract. This paper investigates the use of image-based deep learning methods to automate the segmentation and semantic classification of point clouds for Heritage Building Information Modelling (HBIM). In response to the limitations of classical machine learning approaches such as Random Forests, DBSCAN, and K-Nearest Neighbours, this paper proposes a hybrid pipeline combining 360° panoramic imagery with state-of-the-art computer vision models. The proposed solution leverages Meta’s Segment Anything Model (SAM) for image-based segmentation and YOLO World for open-vocabulary object classification, with segmentation masks reprojected into 3D space to annotate point clouds data of heritage buildings.Experiments were conducted on a high-resolution dataset from the Queen’s House, Royal Museums Greenwich. Results show that SAM generalises well to equirectangular projections, particularly when applied to synthetic panoramas rendered from point clouds. YOLO World enhanced semantic labelling but showed reduced specificity in heritage contexts. The proposed hybrid pipeline produced spatially consistent and semantically enriched 3D segments, demonstrating potential for reducing manual labour in HBIM workflows.Despite challenges related to projection ambiguity, occlusion, and semantic granularity, the research presented in the paper validates a novel paradigm for 3D heritage interpretation that fuses visual intelligence with geometric precision. With the results from the experiment presented in the paper, a future recommendation incorporating multi-view inputs, depth filtering, and ontology mapping is also provided to scale the approach toward practical HBIM adoption.