When the delta is fake: A construction for catching no-op wins in agent evaluation

Technical note and public mini-suite on agent-evaluation craft. Ships three tasks (clearing-hold-ledger, slot-ring-buffer, site-shift-meter) with named starter failures, oracle solutions, verifiers that recompute scores from artifacts, calibration notes, a site-shift case study (why mean-only adaptation can leave ranking AUROC unchanged), and a scored staged-fail trace rubric. Designed against the Terminal-Bench task shape (instruction, environment, outcome tests, oracle) and usable next to Harbor / Inspect-style harnesses. Synthetic fixtures only. Not a leaderboard. Reproduce: pytest -q -p no:randomly tasks/*/tests then python3 tasks/site-shift-meter/solution/train_and_eval.py.

Authors

Institutions

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-10-31
DOI
https://doi.org/10.5281/zenodo.23829461
Primary Topic
Software Testing and Debugging Techniques
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
article

When the delta is fake: A construction for catching no-op wins in agent evaluation

Gyanu Mayank
1 links
Zenodo (CERN European Organization for Nuclear Research)
Software Testing and Debugging Techniques
article

When the delta is fake: A construction for catching no-op wins in agent evaluation

Gyanu Mayank
article en

Abstract

Technical note and public mini-suite on agent-evaluation craft. Ships three tasks (clearing-hold-ledger, slot-ring-buffer, site-shift-meter) with named starter failures, oracle solutions, verifiers that recompute scores from artifacts, calibration notes, a site-shift case study (why mean-only adaptation can leave ranking AUROC unchanged), and a scored staged-fail trace rubric. Designed against the Terminal-Bench task shape (instruction, environment, outcome tests, oracle) and usable next to Harbor / Inspect-style harnesses. Synthetic fixtures only. Not a leaderboard. Reproduce: pytest -q -p no:randomly tasks/*/tests then python3 tasks/site-shift-meter/solution/train_and_eval.py.

Zenodo (CERN European Organization for Nuclear Research)
National Institute of Technology Patna (IN)
Openalex Percentile: Top 3%
Software Testing and Debugging Techniques
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon