A Study on Applying Large Language Models to Issue Classification

Prompt-based large language models (LLMs) have demonstrated their ability to perform tasks with minimal or no additional training data. In the context of issue classification, researchers have actively explored the capabilities of LLMs in classifying issue reports. However, existing studies still face limitations in accuracy. This study replicates an LLM-based issue classification study using GPT-3.5 Turbo and explores variants, such as adopting different models like Llama <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$3.18 B$</tex> and GPT-4o. Experimental results show that the classifier fine-tuned with GPT-3.5 Turbo still yields the same accuracy as shown in the original research and that the classifier fine-tuned with Llama <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$3.18 B(0.8004)$</tex> yields an F1-score of 0.0535 lower than that of the classifier fine-tuned with GPT-3.5 Turbo (0.8467). On the other hand, the classifier with GPT-4o (0.8639) yields an average <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\mathbf{F 1}$</tex>-score <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\mathbf{0. 0 1}$</tex> higher than that of the classifier fine-tuned with GPT-3.5 Turbo (0.8467). Additionally, the project-agnostic classifier fine-tuned with GPT-4o yields the highest <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\mathbf{F 1}$</tex>-score of <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\mathbf{0. 8 6 8 0}$</tex>. These findings contribute to advancing LLM-based issue classification by providing experimental insights into the accuracy of LLMs in this issue classification task.