Stories Complicate Things

Generative AI has found increasing interest in software development, giving rise to coding assistants such as GitHub Copilot. However, the correctness of generated code varies strongly. Objectives. In this study, we explore characteristics of coding problems that could (not) be solved by GitHub Copilot and use our results to point to new research directions. Methods. We use open coding to label 100 LeetCode coding problems and 50 associated solutions. For the coding problems, analyse the impact of the labels on GitHub Copilot’s ability to solve the coding problems. For the solutions, we use the labels to infer general metrics which we subsequently extract for a total of 535 solutions. Results. Our results point to three characteristics leading to coding problems being solved less frequently: (1) Usage of real-world scenarios for explanation, (2) long descriptions and (3) the need for a more complex solution. Conclusion. The results underscore the need for future research to enable LLMs to handle coding problems with a higher complexity. Moreover, further investigation is needed to validate our initial findings regarding a worse performance of LLMs on real-world scenarios in programming.