Does Benchmark Critique Diffuse into Evaluation Practice? A Pilot-Based Study of EvalPlus and SWE-bench+ Adoption

This record is a pilot-study preprint together with its LaTeX source and reproducibility materials. It reports a small, purposive study of whether research papers that cite a benchmark critique go on to use the remediation that critique released. Two remediations are examined. EvalPlus augments HumanEval and MBPP with additional tests; SWE-bench+ releases a filtered 548-instance dataset for SWE-bench. A frozen citation frame of 2,666 title clusters was built from Semantic Scholar records retrieved on 13 November 2026. From that frame, 29 papers were drawn by purposive sampling: 27 full texts were accessible, 12 papers were coded, 8 were eligible for the all-pair set, and 6 for the two primary pairs. Rubric v1.1 distinguishes citation only (0), acknowledgment (1), partial adoption (2), and full adoption (3). One of the six primary-pair papers fully adopted the remediation (16.7%; Wilson 95% CI 3.0–56.4); the all-pair figure is 1 of 8 (12.5%; 2.2–47.1). Three of six primary-pair papers, and four of eight all-pair papers, remediated independently instead of adopting the critique's artifact. The agreement statistics reported in the paper are LLM–LLM (κ = 0.78 unweighted, 0.86 linear weighted) and are not human inter-rater reliability. Status. The paper is a pilot study and a preprint. It has not been peer reviewed and has not been accepted for publication. It does not estimate prevalence in the 2,666-cluster frame, and it does not establish causal diffusion from critique to practice. The full-scale study described in the protocol — a stratified sample of 900 invited papers, human double-coding, and pre-specified sensitivity analyses — was designed but not executed. Contents. paper.pdf (20 pages, A4); paper-source.zip (XeLaTeX source, vector figures and figure data, abstract, reference metadata); reproducibility-package.zip (coding rubric, pilot coding table, analysis scripts and tests, generated tables, protocol, provenance and licensing notes, aggregate frame summary); README.md; LICENSE; LICENSE-DATA. Licensing. The paper, documentation, rubric, and derived data are released under CC BY 4.0. Original code is released under the MIT license. The frozen citation frame and the arXiv metadata export are not redistributed here because they derive from Semantic Scholar records; aggregate counts and SHA-256 hashes are included, and the frame itself is available in the repository under Semantic Scholar's terms. Links. Repository: github.com/P-r-e-m-i-u-m/benchmark-critique-diffusion-protocol. Research website: p-r-e-m-i-u-m.github.io/benchmark-critique-diffusion-protocol.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-15
DOI
https://doi.org/10.5281/zenodo.24244999
Primary Topic
Software Engineering Research
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Does Benchmark Critique Diffuse into Evaluation Practice? A Pilot-Based Study of EvalPlus and SWE-bench+ Adoption

Syed Abdul Aman
Zenodo (CERN European Organization for Nuclear Research)
Software Engineering Research
preprint

Does Benchmark Critique Diffuse into Evaluation Practice? A Pilot-Based Study of EvalPlus and SWE-bench+ Adoption

Syed Abdul Aman
preprint en

Abstract

This record is a pilot-study preprint together with its LaTeX source and reproducibility materials. It reports a small, purposive study of whether research papers that cite a benchmark critique go on to use the remediation that critique released. Two remediations are examined. EvalPlus augments HumanEval and MBPP with additional tests; SWE-bench+ releases a filtered 548-instance dataset for SWE-bench. A frozen citation frame of 2,666 title clusters was built from Semantic Scholar records retrieved on 13 November 2026. From that frame, 29 papers were drawn by purposive sampling: 27 full texts were accessible, 12 papers were coded, 8 were eligible for the all-pair set, and 6 for the two primary pairs. Rubric v1.1 distinguishes citation only (0), acknowledgment (1), partial adoption (2), and full adoption (3). One of the six primary-pair papers fully adopted the remediation (16.7%; Wilson 95% CI 3.0–56.4); the all-pair figure is 1 of 8 (12.5%; 2.2–47.1). Three of six primary-pair papers, and four of eight all-pair papers, remediated independently instead of adopting the critique's artifact. The agreement statistics reported in the paper are LLM–LLM (κ = 0.78 unweighted, 0.86 linear weighted) and are not human inter-rater reliability. Status. The paper is a pilot study and a preprint. It has not been peer reviewed and has not been accepted for publication. It does not estimate prevalence in the 2,666-cluster frame, and it does not establish causal diffusion from critique to practice. The full-scale study described in the protocol — a stratified sample of 900 invited papers, human double-coding, and pre-specified sensitivity analyses — was designed but not executed. Contents. paper.pdf (20 pages, A4); paper-source.zip (XeLaTeX source, vector figures and figure data, abstract, reference metadata); reproducibility-package.zip (coding rubric, pilot coding table, analysis scripts and tests, generated tables, protocol, provenance and licensing notes, aggregate frame summary); README.md; LICENSE; LICENSE-DATA. Licensing. The paper, documentation, rubric, and derived data are released under CC BY 4.0. Original code is released under the MIT license. The frozen citation frame and the arXiv metadata export are not redistributed here because they derive from Semantic Scholar records; aggregate counts and SHA-256 hashes are included, and the frame itself is available in the repository under Semantic Scholar's terms. Links. Repository: github.com/P-r-e-m-i-u-m/benchmark-critique-diffusion-protocol. Research website: p-r-e-m-i-u-m.github.io/benchmark-critique-diffusion-protocol.

Zenodo (CERN European Organization for Nuclear Research)
Software Engineering Research
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon