Evaluating the Effectiveness of Large Language Models in Automated Unit Test Generation

The increasing use of Artificial Intelligence (AI) in software development underscores the need to select suitable Large Language Models (LLMs) for automating software unit test generation. No prior work has been conducted to evaluate the performance of LLM in this domain. To address this gap, this study evaluates the effectiveness of four prominent LLMs—GPT-4, Claude 3.5, Command-R-08-2024 and Llama 3.1—in generating unit test cases. This study particularly aims to evaluate the performance of these models in real-world testing scenarios. Hence, 106 test cases from 23 test suites based on interviews with software experts and QA engineers are used to ensure relevance and comprehensiveness. These test cases are analyzed using JavaScript Engines Specification Tester (JEST) for code coverage and Stryker for mutation testing while adopting both quantitative and qualitative analysis. The findings reveal that Claude 3.5 consistently outperforms the other models against test success rate, statement coverage, and mutation score with the achieved accuracy of 93.33%, 98.01%, and 89.23% respectively. The results also provide insights into the capabilities of LLMs for automated unit test generation and their integration into the continuous software integration pipeline. Further, the findings authenticated the importance of systematically comparing LLMs for test case generation.