Auditable evaluation of behaviors described by security tools

Working preprint (version 1, 30 October 2026). ClaimBench evaluates whether a specific security-tool build exhibits behavior stated in its public description under locked inputs and controls. The study screened 27 behavior units from ten descriptions; fourteen were assessable and executed, four remain pending, eight were excluded, and one was outside scope. Across eight tools in seven families, thirteen bounded criteria were supported and one was refuted. The AI-assisted, result-blind semantic review is a critique of test scope, not a reference standard for publisher meaning. The results do not establish equivalence with entire descriptions, population reliability, or semantic accuracy rates. No independent external replay has been performed. The companion evidence packet is deposited separately with restricted file access because it contains edited inspection derivatives and material from third parties; original bundle manifests do not verify those edited copies. This paper was prepared as an editorial review candidate but has not been submitted to a venue.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-10-30
DOI
https://doi.org/10.5281/zenodo.23814929
Primary Topic
Software Testing and Debugging Techniques
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
preprint

Auditable evaluation of behaviors described by security tools

Federico Pacheco
1 links
Zenodo (CERN European Organization for Nuclear Research)
Software Testing and Debugging Techniques
preprint

Auditable evaluation of behaviors described by security tools

Federico Pacheco
preprint en

Abstract

Working preprint (version 1, 30 October 2026). ClaimBench evaluates whether a specific security-tool build exhibits behavior stated in its public description under locked inputs and controls. The study screened 27 behavior units from ten descriptions; fourteen were assessable and executed, four remain pending, eight were excluded, and one was outside scope. Across eight tools in seven families, thirteen bounded criteria were supported and one was refuted. The AI-assisted, result-blind semantic review is a critique of test scope, not a reference standard for publisher meaning. The results do not establish equivalence with entire descriptions, population reliability, or semantic accuracy rates. No independent external replay has been performed. The companion evidence packet is deposited separately with restricted file access because it contains edited inspection derivatives and material from third parties; original bundle manifests do not verify those edited copies. This paper was prepared as an editorial review candidate but has not been submitted to a venue.

Zenodo (CERN European Organization for Nuclear Research)
Peace, justice, and strong institutions
Software Testing and Debugging Techniques
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon
Auditable evaluation of behaviors described by security tools — Federico Pacheco · Zenodo (CERN European Organization for Nuclear Research) (2026) | TGRS Research Map | TGRS