When the delta is fake: A construction for catching no-op wins in agent evaluation
Technical note and public mini-suite on agent-evaluation craft. Ships three tasks (clearing-hold-ledger, slot-ring-buffer, site-shift-meter) with named starter failures, oracle solutions, verifiers that recompute scores from artifacts, calibration notes, a site-shift case study (why mean-only adaptation can leave ranking AUROC unchanged), and a scored staged-fail trace rubric. Designed against the Terminal-Bench task shape (instruction, environment, outcome tests, oracle) and usable next to Harbor / Inspect-style harnesses. Synthetic fixtures only. Not a leaderboard. Reproduce: pytest -q -p no:randomly tasks/*/tests then python3 tasks/site-shift-meter/solution/train_and_eval.py.
Authors
- Gyanu Mayank
Institutions
- National Institute of Technology Patna (IN)
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-10-31
- DOI
- https://doi.org/10.5281/zenodo.23829461
- Primary Topic
- Software Testing and Debugging Techniques
- Type
- article
- Field-Weighted Citation Impact
- 0.00