Employing Prompt Engineering for Generation of Bug Report: An Experience Report in Industry

The anticipated identification and correction of bugs during software testing processes reduce costs and ensure product quality, preventing these bugs from reaching end users. Consequently, communication between testers and developers regarding existing problems in software occurs through bug reports, which necessitates clear and precise documentation to enable developers to implement corrections effectively. In this context, Large Language Models (LLMs) have been utilized to facilitate bug report composition. This paper presents the results of an empirical study on the application of LLMs to assist in writing bug reports within a real software test team from the industry. To conduct this study, prompts were developed with the objective of employing them to report the most common defects according to the test scope managed by the test team. To evaluate the created prompts, a questionnaire was designed to measure the perception of testers regarding the use of LLMs in generating bug reports. Of the 7 participants from the test team, selected through non-probability convenience sampling, who participated in the study, 71% strongly agreed that utilizing an LLM to report bugs is straightforward, and 86% strongly agreed that it is feasible to use the LLM’s output to create bug reports. Moreover, the participants provided observations regarding the study, such as the occurrence of hallucinations and the necessity to modify the prompt when LLM did not produce the desired response. Thus, this experience report evaluates the implementation of prompts for reporting issues in a test team from the software industry.