Vulnerability prediction using pre-trained models: An empirical evaluation

The rise of Large Language Models (LLMs) has provided new directions for addressing downstream text classification tasks, such as vulnerability prediction, where segments of the source code are classified as vulnerable or not. Several recent studies have employed transfer learning in order to enhance vulnerability prediction taking advantage of the prior knowledge of the pre-trained LLMs. In the current study, different Transformer-based pre-trained LLMs are examined and evaluated with respect to their capacity to predict vulnerable software components. In particular, we fine-tune BERT, GPT-2, and T5 models, as well as their code-oriented variants namely CodeBERT, CodeGPT, and CodeT5 respectively. Subsequently, we assess their performance and we conduct an empirical comparison between them to identify the models that are the most accurate ones in vulnerability prediction.