The Crossover Episode: When Persistent Harness State Pays for Itself

Abstract. Prior work argued that the extra cost of a stateful orchestration harness is an investment recovered at a crossover episode N*, but left N* unmeasured. Concurrent work on efficient agent harnesses reports large token savings from removing redundant reasoning callbacks, with development cost amortised within a single evaluation population. We isolate which mechanism earns the return. We pre-register an eight-episode, three-arm protocol on a deterministic 25-function benchmark with one fixed model (deepseek-flash, hidden reasoning disabled). All arms decompose work into one function per response; they differ only in harness state. F discards state after each episode; R persists verified decisions and executes them through native re-verification in place of a model callback (guarded decision reuse plus certified callback substitution); G adds a proposer, a disjoint acceptance gate and persisted prompt rules --- the stateful RSI team. Two per-response budgets (512 and 256 tokens) were fixed before the runs. Under both budgets the crossover exists and equals N* = 7. The return is almost entirely attributable to decision reuse, not to the RSI layer: at 512 tokens cumulative cost is 9,133 (F), 3,547 (R) and 7,347 (G) tokens, i.e. R saves 61.16% and G saves only 19.56% against F; at 256 tokens the figures are 9,046 / 3,734 / 7,228 and 58.72% / 20.10%. The entire F-to-G difference is the development cost of proposals and paired gate evaluations (3,859 and 3,603 tokens, paid in the first three episodes). Across both conditions the gate accepted zero of six candidate rules, so G's per-episode cost stays within 16% of R's throughout. The pre-registered prediction that gated persistence adds value over reuse alone is therefore not supported on this benchmark. We derive the exact identity N* = min{N : sum_{i<=N}(C_F,i - C_G,i) > C_dev}, show that the observed N* = 7 exceeds the analytical lower bound C_dev / mean(delta C) of about 5.5 because of cold start, and explain why the same mechanism pays off within 96 tasks in a visual-agent report while costing more than half the total budget here: N* is a ratio of development cost to per-task saving, not a property of a method. Artifact. The package contains the pre-registered experiment script, the benchmark and model client, and the two complete raw result files. Every number in the paper is read from those files.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-10
DOI
https://doi.org/10.5281/zenodo.24096723
Primary Topic
Natural Language Processing Techniques
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

The Crossover Episode: When Persistent Harness State Pays for Itself

Jason Wang
Zenodo (CERN European Organization for Nuclear Research)
Natural Language Processing Techniques
preprint

The Crossover Episode: When Persistent Harness State Pays for Itself

Jason Wang
preprint en

Abstract

Abstract. Prior work argued that the extra cost of a stateful orchestration harness is an investment recovered at a crossover episode N*, but left N* unmeasured. Concurrent work on efficient agent harnesses reports large token savings from removing redundant reasoning callbacks, with development cost amortised within a single evaluation population. We isolate which mechanism earns the return. We pre-register an eight-episode, three-arm protocol on a deterministic 25-function benchmark with one fixed model (deepseek-flash, hidden reasoning disabled). All arms decompose work into one function per response; they differ only in harness state. F discards state after each episode; R persists verified decisions and executes them through native re-verification in place of a model callback (guarded decision reuse plus certified callback substitution); G adds a proposer, a disjoint acceptance gate and persisted prompt rules --- the stateful RSI team. Two per-response budgets (512 and 256 tokens) were fixed before the runs. Under both budgets the crossover exists and equals N* = 7. The return is almost entirely attributable to decision reuse, not to the RSI layer: at 512 tokens cumulative cost is 9,133 (F), 3,547 (R) and 7,347 (G) tokens, i.e. R saves 61.16% and G saves only 19.56% against F; at 256 tokens the figures are 9,046 / 3,734 / 7,228 and 58.72% / 20.10%. The entire F-to-G difference is the development cost of proposals and paired gate evaluations (3,859 and 3,603 tokens, paid in the first three episodes). Across both conditions the gate accepted zero of six candidate rules, so G's per-episode cost stays within 16% of R's throughout. The pre-registered prediction that gated persistence adds value over reuse alone is therefore not supported on this benchmark. We derive the exact identity N* = min{N : sum_{i<=N}(C_F,i - C_G,i) > C_dev}, show that the observed N* = 7 exceeds the analytical lower bound C_dev / mean(delta C) of about 5.5 because of cold start, and explain why the same mechanism pays off within 96 tasks in a visual-agent report while costing more than half the total budget here: N* is a ratio of development cost to per-task saving, not a property of a method. Artifact. The package contains the pre-registered experiment script, the benchmark and model client, and the two complete raw result files. Every number in the paper is read from those files.

Zenodo (CERN European Organization for Nuclear Research)
Natural Language Processing Techniques
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon