Decoding building facades : automated element and material recognition in street-level images for large-scale building stock assessments

As a basis for environmental assessments like Life Cycle Assessment (LCA) of large building portfolios, extensive image data of building envelopes must be evaluated as automatically as possible. This paper addresses the automated detection of both elements – e.g., windows, walls, doors – and materials in building facades with unified machine learning workflows using 2D RGB images. Following a systematic review of existing methods and datasets, two unified segmentation workflows are developed: Hierarchical Segmentation (HS) and Multi-Task Learning (MTL). HS exploits the hierarchical relationships between facade elements and materials and deploys a post-prediction clustering approach with Hierarchical Density-Based Spatial Clustering of Applications with Noise (HDBSCAN), while MTL leverages shared feature learning for simultaneous detection. To mitigate limited training data, this work introduces the high-resolution segmentation and classification dataset Facades Material Munich (FaMatMuc). For the first time, element and material detection for facade images were combined in one workflow and validated successfully.