Road2Synth: Reconstructing Pedestrian–Vehicle Encounters from Video

There are many facets of applied science that could benefit from convergent approaches that reach across ethnographic fieldwork, sensing, and simulation in search of unified explanations. This is particularly true in pedestrian and traffic safety science, where empirical traditions for vehicle analysis must often be paired to social science understanding of pedestrian and driver behavior. Alas, these steps are usually pursued in isolation, ceding fidelity of explanation in the gaps in between approaches. There has been growing interest in using video to close these gaps, capitalizing on the prevalence of cameras in fixed traffic infrastructure. This is promising, but fixed infrastructure cameras may miss valuable in-the-moment interactions that occur at eye level as lived experiences of drivers and road-crossers. Here, we introduce Road2Synth, a research pipeline to take cellphone video captured at the roadside and add value to it through a combination of geographic information science, ethnography assisted by deep learning, and reconstructive modeling from extended reality (XR). Road2Synth targets applied analysis of safety in pedestrian–vehicle interactions, supporting statistical modeling from video, as well as establishing the ability to take flagged conflicts from video and replay them from each road user’s unique viewpoint. Road2Synth is designed to be easy to use, as a supplement to traditional forms of traffic survey. We demonstrate a proof of concept for studying right-turn conflicts at signalized crossings, proven on fieldwork across 21 sites, 227 videos, 3668 pedestrian tracks, and 735 vehicle tracks in New York City and its outlying suburbs