A preprocessing framework to assess air temperature and humidity sensor data reliability for urban microclimate analysis

Abstract Urban microclimate analysis helps address challenges such as the urban heat island effect and localised climate variations, which significantly impact environmental sustainability and human well-being in cities. Having access to reliable thermohygrometric measurements is crucial to perform such analysis. This study focuses on air temperature and relative humidity data collected from 119 sensors deployed in the urban area of Padova, Italy, complemented by high-quality reference measurements from two ARPAV (Regional Agency for Environmental Prevention and Protection of the Veneto) meteorological stations. The objective is to establish a robust methodology to identify sensors with compromised data due to non-optimal installation or high dependence on solar radiation. After a preprocessing phase focused on data availability and temporal coverage of each sensor, three statistical tests are performed. In all these tests, the data measured from the meteorological stations serve as the benchmark. The first test evaluates the correlation between sensor measurements and global horizontal irradiance using Pearson, Kendall, and Spearman correlation coefficients. The second test, based on Dynamic Time Warping Distance, compares air temperature data from urban sensors with benchmark temperatures during sunny and cloudy days. The third test is a clustering analysis with Soft Dynamic Time Warping. Results reveal a high degree of consistency among the three tests, all of which identify 29 sensors deviating from expected patterns. Moreover, a combined analysis of these methods, considering cloudy and sunny days separately, allows to understand whether sensor discrepancies arise from solar radiation dependence or other local heat sources, such as the urban heat island effect. This study demonstrates the efficacy of integrating multiple statistical techniques to enhance the reliability of urban sensor networks.