Fine-grained building function recognition with street-view images and GIS map data via geometry-aware semi-supervised learning

The diversity of building functions is vital for urban planning and optimizing infrastructure and services. Street-view images offer rich exterior details, aiding in function recognition. However, street-view building function annotations are limited and challenging to obtain. In this work, we propose a geometry-aware semi-supervised method for fine-grained building function recognition, which effectively uses multi-source geoinformation data to achieve accurate function recognition in both single-city and cross-city scenarios. We restructured the semi-supervised method based on the Teacher–Student architecture into three stages, which involve pre-training for building facade recognition, building function annotation generation, and building function recognition. In the first stage, to enable semi-supervised training with limited annotations, we employ a semi-supervised object detection model, which trains on both labeled samples and a large amount of unlabeled data simultaneously, achieving building facade detection. In the second stage, to further optimize the pseudo-labels, we effectively utilize the geometric spatial relationships between GIS map data and panoramic street-view images, integrating the building function information with facade detection results. We ultimately achieve fine-grained building function recognition in both single-city and cross-city scenarios by combining the coarse annotations and labeled data in the final stage. We conduct extensive comparative experiments on four datasets, which include OmniCity, Madrid, Los Angeles, and Boston, to evaluate the performance of our method in both single-city (OmniCity & Madrid) and cross-city (OmniCity - Los Angeles & OmniCity - Boston) scenarios. The experimental results show that, compared to advanced recognition methods, our method improves mAP by at least 4.8% and 4.3% for OmniCity and Madrid, respectively, while also effectively handling class imbalance. Furthermore, our method performs well in the cross-categorization system experiments for Los Angeles and Boston, highlighting its strong potential for cross-city tasks. This study offers a new solution for large-scale and multi-city applications by efficiently utilizing multi-source geoinformation data, enhancing urban information acquisition efficiency, and assisting in rational resource allocation. • A geometry-aware semi-supervised framework for building function recognition. • A coarse annotation generation module combining street-view images and GIS data. • Notable performance improvements over state-of-the-art with limited labels. • Superior results on both public and new datasets in cross-categorization system.