Roof plane parsing towards LoD-2.2 building reconstruction based on joint learning using remote sensing images

Building models are important for urban studies. Remote sensing multi-spectral (MS) images are widely used for its rich semantic information. The lack of geometry features is fulfilled by introducing photogrammetry derived digital surface models (DSMs), resulting in pairs of DSMs and MS images. Utilizing such pairs and a convolutional neural network, level of detail (LoD) 2.2 building models, which contain roof planes and major roof elements (e.g. dormers), are reconstructed in this work. Leveraging both raster and vector predictions, 3-D building models with straight edges and sharp corners are obtained. The proposed two-stage method first extracts vectorized roof lines from pairs of DSMs and RGB images, followed by generation of detailed 2-D and 3-D polygonal building models. We conducted our experiments based on two datasets: a custom dataset in Landsberg am Lech in Germany, and an open dataset named Roof3D. For the custom dataset, our proposed model achieved mean average precision (mAP) for building roof vertices of 64.3% and for building roof lines of 54.5% at highest. Mean precision and recall for reconstructed 2-D building roof plane polygons are 52.2% and 54.7% respectively. For the Roof3D dataset, mAP is reported to be 25.3% and 12.4% for the extracted building roof lines and roof plane polygons respectively. • We propose a backbone network that fuses geometric and spectral information for detailed building extraction, contributing to the study of DSM and orthophoto pairs. • Our work fills the gap between prediction of roof lines and generation of 3-D building models. • We propose a simple yet effective LoD-2.2 building abstraction as semantic raster prediction which models all kinds of buildings. • We develop an effective workflow of generating LoD-2.2 3-D building models in polygonal format based on RGB images and DSMs.