Flagged on Both Sides: A Paired Before/After Evaluation of Open-Source Static Application Security Testing on Real Vulnerability Fixes

Static application security testing (SAST) tools are scored on synthetic suites, where a vulnerable test case faces an independent safe one. Real vulnerabilities offer a sharper control: the same file after its fix. We build PairCVE, 6,427 before/after file pairs from 4,459 GitHub-reviewed OSV advisories in PyPI, npm, Go, Maven and RubyGems, and score Semgrep CE, CodeQL, Bandit and gosec in seven configurations. A pair counts as detected only when a tool flags the vulnerable file for its weakness class, not the fixed file. Per-file recall ranges from 4.6% to 19.2% (the 19.2% ceiling, gosec, is the least reliable configuration), but for every configuration a majority of the flagged vulnerable files (55-96%) are flagged again after the fix, so paired detection is only 0.9-3.3%. The gap is largest where a rule matches a code pattern the fix does not remove: per-file recall cannot separate a flow-sensitive detection from an intentional pattern match, and the paired view does. Where the fix instead rewrites a sink call (9.9% of pairs), paired detection rises -- Semgrep to 15.6%. Two controls isolate the effect: on the OWASP Benchmark under identical isolation Youden's J exceeds the pairwise J in all 10 tool-suite combinations (sign test p=0.002); whole-repository CodeQL significantly raises recall (recall McNemar p=0.016) but does not change the surviving share (56% and 56%). We argue that per-file recall on real code must be read against its pairwise counterpart, and we release the corpus and code. Files. flagged-on-both-sides.en.pdf (paper, English) · flagged-on-both-sides.fr.pdf (French translation) · flagged-on-both-sides-artifact.zip (reproduction artifact: corpus manifest of 6,427 pairs, scanning and scoring code, pinned tool versions, raw analyzer findings, scores). Reproduce. Unzip the artifact, then run bash flagged-on-both-sides-artifact/experiments/verify_offline.sh. It regenerates every number in the paper from the released scores and fails on any drift (no network, no source files). Source files are not redistributed; the manifest's commit SHAs and SHA-256 hashes let anyone rebuild them from public git history.