Generating Exploratory Test Suites with ChatGPT: Insights from an Empirical Study

Exploratory Testing (ET) plays a critical role in ensuring software quality by uncovering edge cases and enabling adaptive evaluation of systems. However, ET is often constrained by its reliance on tester expertise, challenges in reproducing failures, and limited testing resources. Generative Artificial Intelligence (GenAI), particularly ChatGPT, presents opportunities to support ET by automating tasks such as test case generation. This paper reports on an empirical study comparing exploratory test suites generated by ChatGPT using six image-based prompts with those created by human testers. We evaluated the quality of the generated suites, identified the most effective prompt formats, and gathered feedback from professional testers. Results show that ChatGPT can produce high-quality test suites, though it may miss complex faults and occasionally generate incoherent cases. The best performance was achieved using sub-prompts focused on specific features. Testers found the AI-generated suites clear and practical but noted gaps requiring complementary manual scenarios. These findings suggest that ChatGPT can serve as a valuable aid in ET, while highlighting areas for further improvement.