SchedOps-Bench and Vardiya: a live benchmark and an open harness for language-model agents in batch job scheduler operations
Batch job schedulers run the nightly file transfers, scripts and database loads of many organisations, and their administrators spend much of their time on routine repairs: a rotated password, a broken key permission, a held lock, a full disk, a wrong date pattern in a file name. I present SchedOps-Bench, a live benchmark in which a language-model agent acts as the scheduler administrator in a Docker lab with a real scheduler (JS7), two Linux agent hosts, an SFTP server and a PostgreSQL database. Each scenario really breaks the site, gives the agent a ticket that does not name the root cause, and judges the episode only by the end state read from the servers, with separate verdicts for partial, false green and harmful outcomes. The benchmark has 165 scenarios in six sets, each validated in both directions. I also present Vardiya, an open harness that aims to help small and mid-sized models adapt to this work, and compare three rungs: a bare model (zero by construction), a tool-equipped model with only the access layer, and Usta, the same model inside the full harness. The main measured gain is access: with tools, a light hosted model solves 15 of 15 development episodes. The harness adds step efficiency (8.3 against 14.1 steps on average) and, with a weaker model on the hidden test set, 11 of 27 against 8 of 26 correct with 1 against 4 harmful outcomes, a difference that is not statistically significant. With a small local model (qwen3:14b) on the development set, which the harness was tuned on, the harness removed all harmful outcomes (0 against 10 of 16) and raised correct episodes from 3 to 9. On the hard tier there is no correctness gain: on the 17-scenario hard hidden test set both arms are correct in 0 episodes. I release the lab, scenarios, harness, raw episode evidence and analysis scripts under Apache 2.0, and I present the hard tier as an open challenge.
Authors
- İlhan KOÇASLAN (ORCID: https://orcid.org/0000-0003-4776-9899)
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-11-17
- DOI
- https://doi.org/10.5281/zenodo.24309981
- Primary Topic
- Software System Performance and Reliability
- Type
- preprint