Sidewalk2Synth: generating synthetic embodied locomotion from real-world streetscapes

Abstract We demonstrate methods for generating synthetic scenarios of embodied locomotion on sidewalks, drawn from detailed observations of situation and of context on real-world streetscapes. We show that, through observation and sensing, quite rich and high-resolution data can be gleaned from small and fleeting windows on pedestrian locomotion as it unfolds in lived spaces. These insights can provide valuable explanations of how pedestrians experience and embody encounters in physical and social context at localized and individualized scales of space and time. Using agent AI, we demonstrate that this knowledge can be transferred into high fidelity models, capturing the essence of embodied locomotion and providing a basis for experimentation with what-if scenario as simulation. By implementing simulations as virtual reality media, we showcase an end-to-end experimental pipeline that allows real human participants to embody themselves in synthetic sidewalks, directly using their innate and tangible perception and locomotion. Our approach establishes a new pliability between real and synthetic embodied locomotion, which we argue can provide experimental maneuverability relative to ordinary questions, as well as to extraordinary scenarios that are challenging to examine on the ground. Sidewalk2Synth could also help to circumnavigate existing challenges in machine learning around training-based approaches that lack robust empirical evidence of priors and that are otherwise resistant to generalization outside specific places and times.