Counting Tokens, Not Bugs: Length Confounding and Budget Infeasibility in LLM-Based Vulnerability Detection Evaluation
Reported numbers for LLM-based vulnerability detection on benchmarks such as PrimeVul increasingly justify deployment in triage. We show that the dominant signal behind them is not vulnerability understanding but function length. Within the 435 official vulnerability/fix pairs the two versions of a function differ in length by 0.94 --- fixing a vulnerability does not change a function's length. Across the benchmark's classes the ratio is 5.2 , and token count alone reaches AUROC 0.825 , above a 7B code LLM (0.684), a 73-feature lexical learner (0.814) and a professional analyzer (0.590). We trace the gap to negative-sample construction: 73 % of benign samples are ordinary functions swept in because a security commit touched their file. Three fine-tuned encoders --- the only signals to beat token counting --- reach 0.836--0.849 unmatched but lose 57--68 % of that advantage under 1:1 length balancing (0.618--0.661) and fall below chance on the official paired set, where a code-blind rule that calls the shorter function vulnerable scores 0.829 . We formalize a three-surface diagnostic (unmatched, length-balanced, paired) with a reproducible length-balanced variant of PrimeVul and an attribution ratio, a budget-attainability analysis showing that coverage-calibrated thresholds breach false-positive budgets on quantized scores (4 of 5 operating points under bf16, 0 of 5 under fp32), a ten-item report card, and an attack in which inert dead code collapses a lexical detector's recall from 8.6 % to 0.2 %. A cross-dataset check on Devign shows the confound tracks the sampling rule, and exposes a residual conditional scale channel that marginal matching does not close.
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-11-15
- DOI
- https://doi.org/10.5281/zenodo.24232719
- Primary Topic
- Software Engineering Research
- Type
- preprint