Can a Llama Be a Watchdog? Exploring Llama 3 and Code Llama for Static Application Security Testing

Research in software vulnerability detection has seen significant growth, with numerous systems and techniques being developed. Deep learning approaches have become partic-ularly popular, with various architectures being adapted for this purpose. Llama 3, the newest AI model from Meta, was released in April 2024. It has been trained on an extensive text corpus and contains four times the amount of code compared to its prede-cessor, Llama 2. In contrast, Code Llama stands out as the only model in the Llama series that has been pre-trained specifically on source code. In this study, we examine the effectiveness of the Llama architectures in static security analysis tasks by fine-tuning Llama 3 and Code Llama for vulnerability classification and detection with high precision. To provide comprehensive insights, we compare their performance against three leading models-CodeBERT, PolyCoder, and NatGen-known for their effectiveness in source code analysis, using two benchmark C/C++ datasets.