A Systematic Study on the Potentials and Limitations of LLM-assisted Software Development

In the field of software engineering, Large Language Models like GPT have gained enormous interest in recent times. With its expanding area of application, ChatGPT has become an essential tool for code generation. Several studies have shown that the quality of generated code depends on the underlying dataset and the quality of the provided prompts. However, its precise capabilities and limitations remain uncertain, as does the extent of assistance required for effective code generation. We present the results of our systematic study in which we investigate the potential of ChatGPT, based on GPT-4, in solving assignments of an introductory-level programming class. We examine the impact of programming language choice, different prompting strategies, and the results of the model compared to those of real students. Our results show that ChatGPT cannot solve the assignments independently, but outperforms the average student with human assistance.