Evaluating large language models for software testing

• We present a new evaluation framework for testing the capabilities of LLMs , using manual testing as a benchmark to assess different LLMs. This approach avoids the potential reliance on existing test sets that may be part of LLM training data , ensuring a more reliable evaluation of their generalization capabilities. • We utilize third-party open-source software for LLM testing evaluation, ensuring bug reproducibility and simulating the real-world application of LLMs in software testing, thus enabling more objective evaluations. • We evaluate different LLMs from multiple perspectives and discuss practical strategies for implementing LLM-driven testing effectively. • We propose the follow-up question method for LLM-driven testing, which shows promise in enhancing the abilities of LLMs to detect bugs and defects in program code. Large language models (LLMs) have demonstrated significant prowess in code analysis and natural language processing, making them highly valuable for software testing. This paper conducts a comprehensive evaluation of LLMs applied to software testing, with a particular emphasis on test case generation, error tracing, and bug localization across twelve open-source projects. The advantages and limitations, as well as recommendations associated with utilizing LLMs for these tasks, are delineated. Furthermore, we delve into the phenomenon of hallucination in LLMs, examining its impact on software testing processes and presenting solutions to mitigate its effects. The findings of this work contribute to a deeper understanding of integrating LLMs into software testing, providing insights that pave the way for enhanced effectiveness in the field.