Influence of Large Language Models on Programming Assignments – A user study

This study investigates the influence of chatbots based on general language models, such as ChatGPT, on the work of programmers. We developed a web application called ClueCodeChallenge that allows participants to solve JavaScript programming tasks and interact with a chatbot called ClueBot during the problem-solving process. A user study was conducted with 18 participants aged 19-28, where they completed two tasks, one of which had the option of using the chatbot. The results showed that participants took less time to solve tasks and made fewer errors when they could use the chatbot. The study also found that participants often preferred to ask the chatbot for the complete solution rather than using it as an aid to check their code or get clarification on the programming language. Our next steps are to expand the participant pool and explore the influence of different general language model chatbots on programming tasks across various programming languages.