Segment Anything Model with LiDAR based images for building segmentation in forest areas

Abstract. The Segment Anything Model (SAM) represents a significant advancement in image segmentation, with growing applications for LiDAR-based data alongside traditional RGB imagery. Recent work, such as Ošep et al. (2024) on Segment Anything in LiDAR (SAL) and Yarroudh (2023) on automatic unsupervised LiDAR segmentation with SAM, highlights its potential for enhancing segmentation accuracy in complex environments. Building segmentation, especially in forested areas, poses unique challenges due to difficulties in distinguishing structures from dense vegetation. Prior research indicates that utilizing height information from LiDAR digital surface models (DSM) and digital elevation models (DEM) is beneficial, suggesting SAM could improve forest building segmentation accuracy with LiDAR-based images. This study explores SAM's application for building segmentation using true orthophotos and LiDAR-derived DSMs and DEMs. Its performance is compared against the U-Net neural network (Ronneberger et al. 2015), which utilizes the same multi-modal data. While existing SAM studies often focus on RGB imagery or point clouds, this research specifically investigates its capabilities within challenging forest environments. A 72km2 rural forested area, covering mapsheet L4211D near Karkkila and N3244E near Närpiö, Finland, was selected for testing. Both models were trained using datasets from multiple Finnish cities. Their performance was evaluated using F1-scores during training. For the test areas, which had true orthophotos, LiDAR DSMs, and DEMs from 2024, the number of correctly identified buildings was analyzed against the topographic database of Finland (1,380 buildings in Karkkila, 1,020 in Närpiö). Additionally, the shape and accuracy of segmented buildings were visually compared. This evaluation of SAM’s effectiveness aims to advance methodologies for building extraction in forested landscapes, ultimately seeking to reduce manual labor in future mapping tasks.