Towards a Dataset to Assess Conversational Agent’s Efficacy in Mentoring Newcomer Developers

Many Open Source Software (OSS) maintainers experience burnout due to the constant need to support and answer questions posed by newcomer developers. This leads to slower response times, and unanswered questions which discourage newcomers and hinder community growth. Research indicates that conversational agents trained on community data can answer common questions, thus reducing maintainers’ work-load. However, conversational agents should only be used if they are useful to newcomers. To help prove a conversational agent can be useful, we manually curated a dataset of questions posted to the JabRef issue tracker and Gitter, submitted by newcomers. This dataset of questions will be invaluable for conducting studies using LLMs to support newcomers. This dataset paper details our collection process, describes the data, and offers reflections on future research.