Urban road surface condition detecting and integrating based on the mobile sensing framework with multi-modal sensors

Abstract The urban road surface condition, encompassing elements such as road surface damages and facility distributions, which collectively refers to bump features of road surface (BFRS), is critical for road maintenance and management in smart cities. Traditional BFRS collection methods relying on specialized equipment are often time-consuming and labor-intensive, whereas crowdsourced data frequently suffers from inconsistent quality and reliability. To address these issues, the BFRS is detected and integrated via a mobile sensing framework (MSF), which leverages multi-modal Internet of Things sensors, including global positioning system, gyroscope, accelerometer, camera, and a 4G module. First, the MSF was designed, and a comprehensive set of multi-modal sensors was seamlessly integrated into a Jetson Nano, and further deployed on the vehicle. Second, the software for BFRS computing was implemented to process the collected multi-modal sensor data. This software comprises three key components: the collection of multi-modal sensor data at the mobile end, data processing on the server side, and map interaction and visualization via the web browser. Finally, the fine-tuned TimesFM model (FTTM) dedicated to BFRS acquisition was proposed. BFRS were detected using spatially transformed acceleration data derived from the Inertial Measurement Units. These detections were temporally synchronized and integrated with corresponding road images identified by a specially trained YOLO model. Experiments were conducted, and the BFRS was detected and integrated using the proposed FTTM, with results presented through web visualizations. The results demonstrated that the proposed method effectively captures the road surface condition using the MSF, highlighting its significant potential for applications in smart cities.