SoK: Fired, Not Fed - A Measurement-Validity Protocol for Obfuscation-Strength Benchmarking in Neural Decompilation

Security evaluations that pit a code-transforming defense against an AI attacker typically report one number: how far the defense drove the attacker's success rate down. We systematize that number in a setting where its terms can still be measured separately, some by re-reducing per-item records, others by building an explicit control. The neural-decompilation and deobfuscation subfield reports obfuscation strength as a single scalar: the drop in a decompiler's re-executable recovery between clean and obfuscated binaries. That scalar is structurally un-auditable. Because the per-function rate at which each transform fired goes unpublished, the aggregate drop confounds five effects: coverage/selection, truncation, a compiler-provenance shift, a disassembler-format shift, and attacker-model conditioning. The same obfuscated binary recovers at ~0.9% against an obfuscation-naive attacker but 12-16% against a fairly-trained obfuscation-aware one. Across a 96-paper survey, 83 of 91 full-text papers report a bare aggregate scalar, and of the 20 evaluations for which the per-function firing rate is decisive, exactly one publishes it. We propose a five-plank protocol: recovery reported fired-only, truncation-stratified, with confidence intervals, decomposed into a provenance-shift term versus a genuine-semantic-hardening residual, and evaluated against a fairly-trained obfuscation-aware (Kerckhoffs) attacker or with the attacker's training coverage labeled. In our controlled study, generic IR-level obfuscation drives a provenance-matched but obfuscation-naive attacker's fired-only recovery to ~0% off a modest 18.8% clean base, while a Kerckhoffs attacker reads the identical configs through at 12-16%. This floor is set by attacker-training coverage and bounds deterrence against the obfuscation-naive attacker only; it is never "unbreakable."