Vulnerability Detection Based on Pre-trained Code Language Model and Convolutional Neural Network

Software vulnerabilities damage the reliability of software systems.Recently, many methods based on deep learning have been proposed for vulnerability detection by learning features from code sequences or various property graphs.However, sequence-based methods frequently fail to consider the structural information of the code, whereas graph-based methods often encounter difficulties in capturing long-distance contextual semantic information.To overcome these limitations, we propose PTCN, a novel vulnerability detection method that integrates Pre-Trained code language model and Convolutional Neural Network (CNN).First, the code tokens sequence and the propagation chains are constructed from the source code in preparation for the subsequent feature extraction.Next, GraphCodeBERT, a graph-based pre-trained code language model, is introduced as a means of learning both the semantic and structural information contained in the code tokens sequence and the propagation chains.Then, a CNN architecture is designed to effectively extract the crucial vulnerability features from the hidden states of the last Transformer layer of GraphCodeBERT for vulnerability detection.The experiments on the public benchmark dataset REVEAL indicate that PTCN outperforms the state-of-the-art methods with respect to the Accuracy and F1-score metrics by 3.41%-15.80%and 22.68%-104.27%,respectively.