Leveraging AI-based Agents to Assist in Experimentation in Software Startups : A Study on Video-Game Software Startups

Background. Startups often face pressure to innovate quickly, which leads to the launch of products with high uncertainty about user expectations and market fit. These uncertainties are based on assumptions that, if not validated, can result in poor decision-making and ultimately product failure. Experimentation helps startups to test these ideas and assumptions early, hereby reducing the risk and enabling evidence-driven development. However, due to limited time, resources, and structure, many startups, particularly in the video game industry struggle to adopt effective experimentation practices. Objectives. With the adoption of Generative AI into software development processes, this study aims to explore how a GenAI agent can assist startups in carrying out experimentation. It aims to understand startups’ perceptions of GenAI, identify relevant experimentation practices and data sources, and finally, develop and evaluate a GenAI agent that is tailored to support video game startups in experimentation Methods. This study employed the Design Science Research (DSR) methodology. First, a survey of video game startups was conducted to identify key experimentation practices, challenges, data sources, and perceptions of Generative AI. Based on the results, a GenAI agent was developed using a Retrieval-Augmented Generation (RAG) architecture, and it was scoped to support technical prototyping in the adventure game genre. Finally, the agent was evaluated by eight participants through semi-structured interviews to assess its usefulness and applicability in early-stage experimentation. Results. Findings from this study show that GenAI is perceived as a helpful support tool for experimentation practices such as technical prototyping, pitching of game ideas, and social media engagement. The most relevant data sources identified for supporting experimentation include Reddit, Steam forums, LinkedIn, and game-specific blogs. Also, the evaluation feedback indicated that the GenAI agent can assist in experimentation to save time, improve decision-making, and it is especially effective in the early prototyping phase. However, it was discovered that its value would be limited in experimentation practices that require subjective or human-centered feedback, such as the release of a vertical slice and controlled game tests. Conclusions. This thesis demonstrates the practical application of a GenAI agent to support experimentation in video game startups. The agent addressed several known inhibitors to experimentation, such as lack of time and limited access to relevant data. It also lays the groundwork for expanding the application of GenAI agents to support additional game genres and experimentation practices.