Using street view images and visual LLMs to predict heritage values for governance support: risks, ethics, and policy implications

Abstract Until 2026, the Energy Performance of Buildings Directive is implemented in the European Union member states, requiring all member states to have National Building Renovation Plans. In Sweden, there is no comprehensive national register of buildings with heritage values. The purpose of this research was to assist Swedish authorities in developing comprehensive information on heritage values. Buildings in street view images from all over Sweden ( N = 154,710) have been analysed using multimodal Large Language Models (LLM) to assess visible aspects indicative of heritage value. Zero-shot predictions by LLMs were used for identifying buildings with potential heritage values for 5.0 million square meters of heated floor area. The results of the predictions and lessons learned are related to the development of the Swedish Building Renovation Plan as part of governance. Risks with authorities’ use of LLM-based data are addressed, with a focus on issues of transparency, error detection and sycophancy.