Analysis of the Effectiveness of Large Language Model Feature in Source Code Defect Detection

The defects in software source code threaten the usage of software. With the increase in the quantity and complexity of software, traditional methods of source code defect detection are unable to meet the needs of software security testing, leading to the emergence of machine learning-based source code defect detection methods. The effectiveness of machine learning-based method is closely related to the input features. This paper investigates the impact of using outputs from large language model as input features for machine learning models on the task of source code defect detection. Experimental results demonstrate that adding feature from large language model to machine learning models can enhance the effectiveness of source code defect detection. Furthermore, among different strategies for constructing prompt words, few-shot prompting shows relatively better performance.