Receipt-Chained Evaluation: Reproducible Benchmark Claims from Raw Model Replies
Benchmark numbers in machine-learning papers are usually reported as aggregates whose provenance stops at a table. We describe an evaluation artifact in which every reported figure is a function of a hash-chained log of raw model replies, an item bank, and a deterministic derivation that a reader can rerun offline. We make three contributions: (i) a receipt format that binds each run to the exact item bank it was scored against; (ii) a correctness definition that re-parses the raw reply label rather than trusting a harness-side flag, and the documented disagreement this produces; and (iii) a verification target that fails the build when any derived number, chain hash, or item-bank hash does not match. We report what the verifier covers, what it deliberately does not, and the failure modes we found while making the bundle cold-clone clean. Artifact. This record contains the compiled paper (paper/main.pdf), the full LaTeX source, the item banks, the hash-chained receipt log, derive.py and verify.py, and a deterministic bundle.sh that rebuilds and re-verifies the bundle from a cold clone. Run make verify to check every derived number against the receipts. Note on Table 1. The shipped receipts are a synthetic fixture that exercises the mechanism end to end; they are not an empirical benchmark result. The paper says so under the table. Paper text and figures: CC BY 4.0. Code: MIT.
Authors
- Ali Shuaib
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-11-14
- DOI
- https://doi.org/10.5281/zenodo.24223927
- Primary Topic
- Scientific Computing and Data Management
- Type
- preprint