Poster: AI-Driven Security: Investigating LLMs for Automated Vulnerability Detection in Code Changes

Security vulnerabilities in software systems give rise to serious threats nowadays because of data breaches, illegal access, and system breakdowns. Traditional vulnerability detection techniques, such as manual code inspections and Static Application Security Testing (SAST) technologies, have been extensively used to find security holes. These methods are good at finding known vulnerabilities, but they frequently follow preset guidelines, which limits their ability to recognize complex or situation-specific security threats.As artificial intelligence has progressed, Large Language Models (LLMs) have become a viable substitute for automated security analysis. In this work, we assess how well LLMs identify vulnerabilities in pull requests (PRs) by contrasting their results with those of Bandit, a popular SAST tool. We focus on free and pre-trained open-source LLM model - ‘Llama 3.2’ using the ‘Ollama’ tool to query the output.The findings show that Llama3.2 with the Rubric generates security assessments that closely match Bandit’s conclusions more when using the rubric compared to without it. By looking at how these models work in different situations, this study tries to find out if LLMs can be used along with or instead of traditional security methods to find vulnerabilities. The results offer a thorough assessment of LLM-based security analysis, including its advantages, disadvantages, and possible real-world uses. They also shed light on the developing role of AI in software security.