Counting Tokens, Not Bugs: Length Confounding and Budget Infeasibility in LLM-Based Vulnerability Detection Evaluation

Reported numbers for LLM-based vulnerability detection on benchmarks such as PrimeVul increasingly justify deployment in triage. We show that the dominant signal behind them is not vulnerability understanding but function length. Within the 435 official vulnerability/fix pairs the two versions of a function differ in length by 0.94 --- fixing a vulnerability does not change a function's length. Across the benchmark's classes the ratio is 5.2 , and token count alone reaches AUROC 0.825 , above a 7B code LLM (0.684), a 73-feature lexical learner (0.814) and a professional analyzer (0.590). We trace the gap to negative-sample construction: 73 % of benign samples are ordinary functions swept in because a security commit touched their file. Three fine-tuned encoders --- the only signals to beat token counting --- reach 0.836--0.849 unmatched but lose 57--68 % of that advantage under 1:1 length balancing (0.618--0.661) and fall below chance on the official paired set, where a code-blind rule that calls the shorter function vulnerable scores 0.829 . We formalize a three-surface diagnostic (unmatched, length-balanced, paired) with a reproducible length-balanced variant of PrimeVul and an attribution ratio, a budget-attainability analysis showing that coverage-calibrated thresholds breach false-positive budgets on quantized scores (4 of 5 operating points under bf16, 0 of 5 under fp32), a ten-item report card, and an attack in which inert dead code collapses a lexical detector's recall from 8.6 % to 0.2 %. A cross-dataset check on Devign shows the confound tracks the sampling rule, and exposes a residual conditional scale channel that marginal matching does not close.

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-15
DOI
https://doi.org/10.5281/zenodo.24232719
Primary Topic
Software Engineering Research
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Counting Tokens, Not Bugs: Length Confounding and Budget Infeasibility in LLM-Based Vulnerability Detection Evaluation

Zenodo (CERN European Organization for Nuclear Research)
Software Engineering Research
preprint

Counting Tokens, Not Bugs: Length Confounding and Budget Infeasibility in LLM-Based Vulnerability Detection Evaluation

preprint en

Abstract

Reported numbers for LLM-based vulnerability detection on benchmarks such as PrimeVul increasingly justify deployment in triage. We show that the dominant signal behind them is not vulnerability understanding but function length. Within the 435 official vulnerability/fix pairs the two versions of a function differ in length by 0.94 --- fixing a vulnerability does not change a function's length. Across the benchmark's classes the ratio is 5.2 , and token count alone reaches AUROC 0.825 , above a 7B code LLM (0.684), a 73-feature lexical learner (0.814) and a professional analyzer (0.590). We trace the gap to negative-sample construction: 73 % of benign samples are ordinary functions swept in because a security commit touched their file. Three fine-tuned encoders --- the only signals to beat token counting --- reach 0.836--0.849 unmatched but lose 57--68 % of that advantage under 1:1 length balancing (0.618--0.661) and fall below chance on the official paired set, where a code-blind rule that calls the shorter function vulnerable scores 0.829 . We formalize a three-surface diagnostic (unmatched, length-balanced, paired) with a reproducible length-balanced variant of PrimeVul and an attribution ratio, a budget-attainability analysis showing that coverage-calibrated thresholds breach false-positive budgets on quantized scores (4 of 5 operating points under bf16, 0 of 5 under fp32), a ten-item report card, and an attack in which inert dead code collapses a lexical detector's recall from 8.6 % to 0.2 %. A cross-dataset check on Devign shows the confound tracks the sampling rule, and exposes a residual conditional scale channel that marginal matching does not close.

Zenodo (CERN European Organization for Nuclear Research)
Software Engineering Research
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon