RE-Bench Is a Systems Benchmark: What Its Scorers and Selection Rules Actually Support

This version-bounded technical note examines what RE-Bench (Research Engineering Benchmark, V1) directly measures and which inferences its scores can support. It treats the evaluated unit as a complete agent system—including the model, scaffold, tools, execution environment, feedback surface, hardware, time budget, attempt allocation, and selector—rather than as a base model in isolation. The review keeps two evidence surfaces separate: results and methods reported in the historical RE-Bench paper, and properties of the public implementation at pinned METR/RE-Bench commit 93b98062e55f6945d4a7e213a3226dd419896170. All seven task packages at that revision were statically inspected. A published eight-test harness additionally checks bounded behaviors in four hash-pinned files across Optimize a Kernel, Scaling Law Experiment, and Optimize LLM Foundry. The three detailed cases illustrate distinct measurement boundaries: selected single-call timing and public contract drift in Kernel; a static eleven-point evaluator and one-sided FLOP gate in Scaling Law; and a declared L1 quantity implemented as a sum of per-tensor norms in LLM Foundry, where parameter proximity remains narrower than behavioral equivalence. The note also distinguishes paper-level score@k selection from single-run reliability, within-run aggregation, and an agent’s ability to identify its own best artifact. This is an unaffiliated, AI-assisted pinned-source contract audit. It is not peer reviewed, independently adjudicated, or an independent benchmark-hardware or headline-result reproduction. Exact mapping between the pinned source and the historical paper-run environments remains unresolved; consequently, the review makes no claim that the identified source properties changed, invalidated, or explain any historical leaderboard result.