Building Extraction From Multi-View RGB-H Images With General Instance Segmentation Networks and a Grouping Optimization Algorithm

Bird’s-Eye-View (BEV) building mapping from remote sensing images is a studying hotspot with broad applications. In recent years, deep learning has significantly advanced the development of automatic building extraction methods. However, most existing research focuses on segmenting buildings from a single perspective, such as orthophotos, overlooking the rich information of multi-view images. In surveying and mapping, individual building instances need to be separated even when they are adjacent or touching. Since orthophotos cannot capture building walls due to self-occlusion, distinguishing between closely connected buildings in densely built areas becomes challenging. To tackle this issue, we propose a multi-view collaborative pipeline for instance-level building segmentation. This pipeline utilizes a grouping optimization algorithm to merge segmentation results from multiple views, which are predicted by general instance segmentation networks and projected onto the BEV, to produce the final building instance polygons. Both qualitative and quantitative results show that the proposed multi-view collaborative pipeline significantly outperforms the popular orthophoto-based pipeline on the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">InstanceBuilding</i> dataset.