Use of ChatGPT as an Assistant in the End-to-End Test Script Generation for Android Apps

Automated testing is crucial in software development to ensure that applications perform as intended. However, generating automated End-to-End (E2E) tests can be time-consuming and challenging, especially for junior developers. This study investigates the use of ChatGPT, a popular Generative Artificial Intelligence (GenAI) model, as an assistant in developing automated E2E test scripts for Android apps. We present an empirical study that compares the effort required to create E2E test scripts and the resulting reliability of these tests using two treatments: manually and assisted by ChatGPT. We used Gherkin, a domain-specific language that allows non-technical practitioners to define test scenarios using a human-readable syntax. Our findings indicate that using ChatGPT significantly reduces the time required to develop automated test scripts without compromising the reliability of the scripts. Statistical analysis shows a notable reduction in development time for the ChatGPT-assisted group compared to the manual group, with a large effect size. While the reliability of the tests did not show a significant difference between the two groups, the results suggest practical benefits in terms of efficiency.