Can a small open model learn to detect code vulnerabilities? A QLoRA study with honest negative results

QLoRA fine-tuning on real vulnerability data gives a small open model a clear, transferable vulnerability-detection advantage. Our 7B adapter (Qwen2.5-7B-Instruct trained on PrimeVul) reaches selective AUROC 0.786 on 200 balanced held-out PrimeVul cases, up from 0.658 for its base model (+0.128), and ranks first among all models scored in the same harness, ahead of a code-specialised model (Qwen2.5-Coder-7B, 0.664) and a security-specialised model (Foundation-Sec-8B, 0.550). It is also far better calibrated (ECE 0.047 against 0.142). The advantage transfers to the unseen DiverseVul corpus: AUROC rises from 0.578 to 0.687 (+0.109) and calibration error falls from 0.185 to 0.034, and across four test-set resamples the mean AUROC is 0.698 against 0.604 with no overlap on any metric. The same recipe improves on its base at every size tested: +0.105 at 1.5B, +0.128 at 7B, +0.033 at 14B and +0.120 at 32B. Every number is recomputed from recorded per-case outputs by an automated integrity check, and the code, adapters and per-case results are public. Code, evaluation harness and per-case results: github.com/pallav0107/cyberdef-llm. Model (7B adapter): huggingface.co/pallav0101/cyberdef-vuln-qwen2.5-7b-primevul. AI assistance. The author used AI assistants (Anthropic's Claude and Moonshot's Kimi, via a coding-agent harness) to write and run the experiment code, generate the result tables, and draft and edit this manuscript. The author directed the study and reviewed all of its content. Every reported number is recomputed from recorded per-case outputs by an automated integrity check published in the repository, and the author takes full responsibility for the paper.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-13
DOI
https://doi.org/10.5281/zenodo.24206850
Primary Topic
Software Engineering Research
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Can a small open model learn to detect code vulnerabilities? A QLoRA study with honest negative results

Pallav Pokhriyal
5 links
Zenodo (CERN European Organization for Nuclear Research)
Software Engineering Research
preprint

Can a small open model learn to detect code vulnerabilities? A QLoRA study with honest negative results

Pallav Pokhriyal
preprint en

Abstract

QLoRA fine-tuning on real vulnerability data gives a small open model a clear, transferable vulnerability-detection advantage. Our 7B adapter (Qwen2.5-7B-Instruct trained on PrimeVul) reaches selective AUROC 0.786 on 200 balanced held-out PrimeVul cases, up from 0.658 for its base model (+0.128), and ranks first among all models scored in the same harness, ahead of a code-specialised model (Qwen2.5-Coder-7B, 0.664) and a security-specialised model (Foundation-Sec-8B, 0.550). It is also far better calibrated (ECE 0.047 against 0.142). The advantage transfers to the unseen DiverseVul corpus: AUROC rises from 0.578 to 0.687 (+0.109) and calibration error falls from 0.185 to 0.034, and across four test-set resamples the mean AUROC is 0.698 against 0.604 with no overlap on any metric. The same recipe improves on its base at every size tested: +0.105 at 1.5B, +0.128 at 7B, +0.033 at 14B and +0.120 at 32B. Every number is recomputed from recorded per-case outputs by an automated integrity check, and the code, adapters and per-case results are public. Code, evaluation harness and per-case results: github.com/pallav0107/cyberdef-llm. Model (7B adapter): huggingface.co/pallav0101/cyberdef-vuln-qwen2.5-7b-primevul. AI assistance. The author used AI assistants (Anthropic's Claude and Moonshot's Kimi, via a coding-agent harness) to write and run the experiment code, generate the result tables, and draft and edit this manuscript. The author directed the study and reviewed all of its content. Every reported number is recomputed from recorded per-case outputs by an automated integrity check published in the repository, and the author takes full responsibility for the paper.

Zenodo (CERN European Organization for Nuclear Research)
Software Engineering Research
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon