On the Need for Reproducibility Guidelines for Open-Source Games: A itch.io Case Study

Video games represent a unique type of complex software project, despite differing in aspects such as methodologies, programming languages, and design patterns. Similar to traditional software, games can be released as open-source software (OSS) projects, thus fostering collaboration and knowledge sharing among developers. However, one of the primary challenges in this domain is the difficulty in finding source code for game development, which is often scattered across various repositories and platforms. Moreover, the lack of proper guidelines to document game projects is still missing, thus worsening this issue. In this paper, we envision a set of initial guidelines leveraging a mining-based methodology considering two different OSS platforms, i.e. itch.io and GitHub. First, we collect data from 765 open-source games from the itch.io platform and map the retrieved games to the corresponding repositories on GitHub, searching for documentation and source code. We further refine the list of games by manually analyzing the repositories, focusing on the quality of the documentation and the presence of source code, ending up with 613 games. On top of this gold set, we elicited a set of ten reproducibility guidelines specifically tailored for games. Our results show that the majority of the games do not have source code available, and the documentation quality is generally low, apart from games with high-rated GitHub projects. In addition, we provide a set of takeaways that can be further investigated by extending the provided guidelines. We believe that our dataset and methodology can be used as a starting point for future research in this domain, providing insights into the challenges and opportunities for assisting newcomers to game development using open-source projects.