Code sharing and reproducibility in survey-based social research: evidence from a large-scale audit

Abstract Reproducibility—the ability to obtain original results by reapplying the original analyses to the original data—is an essential component of empirical research. In this study, we assess the reproducibility of articles using the European Social Survey (ESS), a large-scale repeated cross-sectional dataset widely used across the social sciences. Drawing on more than 1000 ESS-based articles published between 2015 and 2020, we investigate whether authors share their code for reproduction purposes and whether published results are reproducible. We find that only about one in three authors (35%) share code. From the articles with code, we randomly selected 100 which reported 699 results. Of these 699 results, about half (51%) are numerically reproducible, while the others either fail (23%) or are different (26%). For those that are different, numerical deviations are usually minor and do not indicate systematic bias. Overall, about one in six published results (18%) is exactly reproducible. Reproducibility differs somewhat between disciplines, but reproducibility problems persist throughout. Reproducibility failure mostly stems from unavailable, poorly documented, or incomplete code. We propose low-cost measures for authors, editors, journals and data providers to improve code availability and reproducibility in large-N observational social research.