Observing bots in the wild: A quantitative analysis of a large open source ecosystem

The GitHub platform allows repository maintainers and contributors to automate their activities either through GitHub Actions workflows or by making use of bots. The second category concerns automated actors that interact with regular users in GitHub repositories, and includes GitHub’s internal automation services (e.g., Dependabot), GitHub Apps, and bot accounts. There has been little to no empirical research on the presence and use of such bots in large software ecosystems. Also, no former studies have compared how bot accounts and GitHub Apps are used within and across GitHub repositories. This paper addresses this gap, through an empirical analysis of the use of bot accounts and GitHub Apps in NumFOCUS, a large open source software ecosystem for data science. We analyse, over a three-month period, the activity sequences of 853 contributors across 59 GitHub organisations hosting 1,169 repositories. Using state-of-the-art bot identification approaches we identify activities of 802 humans, 34 bot accounts, 13 GitHub Apps and 4 internal automation services. Based on this dataset we reveal behavioural differences in NumFOCUS between bots and humans on the one hand, and between different bot categories on the other hand.