Exploring Fine-Grained Bug Report Categorization with Large Language Models and Prompt Engineering: An Empirical Study

Accurate classification of issues is essential for effective project management and timely responses, as the volume of issue reports continues to grow. Manual classification is labor-intensive and error-prone, necessitating automated solutions. While large language models (LLMs) show promise in automated issue labeling, most research focuses on broad categorization (e.g., bugs, feature requests), with limited attention to fine-grained categorization. Understanding specific bug types is crucial, as different bugs require tailored resolution strategies. This study addresses this gap by evaluating LLMs and prompt engineering strategies for fine-grained bug report categorization. We analyze 221,184 fine-grained bug report category labels generated by selected LLMs using various prompt engineering strategies for 1,024 bug reports. We examine how LLMs and prompt engineering influence output characteristics, control over outputs, and categorization performance. Our findings highlight that LLMs and prompt engineering significantly impact output consistency and classification capability, with some yielding consistent results and others introducing variability. Based on these findings, we analyze the agreements and disagreements between LLM-generated labels and human annotations to assess category correctness. Our results suggest that examining label consistency and discrepancies can serve as a complementary method for validating bug report categories, identifying unclear reports, and detecting misclassifications in human annotations.