Enabling Semantic-Rich Location Search on Street View Imagery Using Multilingual POI Data

Street view imagery is a valuable resource for understanding the physical environment and public health. Open-access street view imagery platforms offer street view imagery along with metadata, such as geographic coordinates and capture details (e.g., the times-tamp of when the photo was taken and camera product type). Metadata improves the utility of street view imagery for various location-based tasks, including urban studies and geospatial search. Recent research has further improved street view imagery by incorporating additional metadata, including human perceptions, weather conditions, and seasonal details. However, connecting street view imagery with Points of Interest (POI) data, including their names and amenity types, remains challenging. In this paper, we propose MulMapper, an automated system that enhances street view imagery metadata by extracting multilingual text, identifying text alignment, and matching POI attributes to the corresponding entity in the image. We develop two modules on top of one of the existing text spotter models: (1) a 'text alignment detector' to capture text alignment types and (2) a 'character-wise text classification loss' to overcome long-tail recognition issues, which result from unbalanced data distribution across diverse character sets. The proposed method greatly enhances the accuracy of matching between POIs and street view imagery while also enabling more semantically rich location searches within the images.