Auditable evaluation of behaviors described by security tools
Working preprint (version 1, 30 October 2026). ClaimBench evaluates whether a specific security-tool build exhibits behavior stated in its public description under locked inputs and controls. The study screened 27 behavior units from ten descriptions; fourteen were assessable and executed, four remain pending, eight were excluded, and one was outside scope. Across eight tools in seven families, thirteen bounded criteria were supported and one was refuted. The AI-assisted, result-blind semantic review is a critique of test scope, not a reference standard for publisher meaning. The results do not establish equivalence with entire descriptions, population reliability, or semantic accuracy rates. No independent external replay has been performed. The companion evidence packet is deposited separately with restricted file access because it contains edited inspection derivatives and material from third parties; original bundle manifests do not verify those edited copies. This paper was prepared as an editorial review candidate but has not been submitted to a venue.
Authors
- Federico Pacheco (ORCID: https://orcid.org/0000-0002-2803-1162)
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-10-30
- DOI
- https://doi.org/10.5281/zenodo.23814929
- Primary Topic
- Software Testing and Debugging Techniques
- Type
- preprint