Revisiting reproducibility in transportation simulation studies

Abstract Most transportation publications to date lack sufficient transparency in terms of open source and data, in order to enable reproducibility. This work sets out to highlight the need for reproducibility of computational simulation studies, and to develop the level of professionality in the transportation domain when it comes to open source. Moreover, this work aims to equip the readers with a guide on best practices inspired by the computer science domain, templates for documentation of software and data, and guidelines on repository management and open source. The empirical analysis of more than 11,879 simulation studies and 672 repositories in the field reveals that only few works (less than 2%) share supplementary material alongside the scientific publication, on a mostly mediocre repository quality, but with a strong trend towards more transparency in the last years. Besides, simulation studies with enhanced transparency are found to not receive significantly more citations when compared with studies that do not provide additional repositories. What’s more, a survey of more than 87 transportation researchers conveys that reproducibility is perceived as a significant issue by the community, and that time and legal constraints are the largest impediments to more reproducible research.