The Orchestrator Adds Nothing on Recovery: A Pre-registered Paired Study of LangGraph versus a Native Loop on Success, Cost, and Exactly-Once Resume

Agent frameworks promise durable, resumable agents, and the promise is rarely tested against a plain hand-written loop under matched controls. This study pre-registered the test. Two arms run the same 12 tasks with the same deterministic scripted policy, the same tools, the same step executor, and the same budgets, paired by seed: one arm sequences the work with LangGraph 1.2.20 and its SQLite checkpointer, the other with a hand-written loop over its own SQLite event log. The scripted policy is a declared scope limit: the study measures the orchestration layer only (sequencing, durability, recovery, re-execution, cost, code surface), not model quality. On task success the arms are equivalent with evidence: overall pass^1 is 0.979 in both arms (94 of 96 runs each), the mean paired per-task difference is 0.000 with a 90 percent confidence interval of [0.000, 0.000], inside the registered equivalence margin of plus or minus 0.05 (H1: Supported). Consistency (pass^k, k = 1 to 8) is identical in the two arms at every k. On cost, policy invocations are identical (median 5.0 per run in both arms, ratio 1.000 against a registered bound of 1.05) and LangGraph's median wall time per run is lower in these implementations (0.293 seconds against 0.980, ratio 0.299 against a bound of 1.10), a comparison the paper flags as implementation-sensitive because the native arm commits state synchronously after every step (H2: Supported). On recovery after a SIGKILL (48 fault runs per arm, two kill classes), both arms pass every run with every intended side effect applied exactly once and no duplicates: the exactly-once resume rate is 1.000 in both arms, so the registered tie form holds and the orchestrator adds nothing measurable on recovery in this design (H3: Supported, tie form). The difference is in what recovery costs: in the step-boundary kill class, LangGraph recomputed exactly one already-finished step in all 24 runs (its durable checkpoint lags the last finished step), while the native event log recomputed none in all 24; LangGraph also needed less arm-specific code (81 against 96 code lines) and a durable state store about 2.25 times larger (median 36,864 against 16,384 bytes per run). Exactly-once came from the idempotency discipline both arms applied, not from the orchestrator. The debugging axis (H4) is reported as Not yet measured, for the reason registered in advance: the arms persist traces in incommensurable units, so the registered record-count procedure would measure storage format, not diagnosis effort.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-18
DOI
https://doi.org/10.5281/zenodo.24328208
Primary Topic
Scientific Computing and Data Management
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

The Orchestrator Adds Nothing on Recovery: A Pre-registered Paired Study of LangGraph versus a Native Loop on Success, Cost, and Exactly-Once Resume

Jesun Ahmad Ushno
6 links
Zenodo (CERN European Organization for Nuclear Research)
Scientific Computing and Data Management
preprint

The Orchestrator Adds Nothing on Recovery: A Pre-registered Paired Study of LangGraph versus a Native Loop on Success, Cost, and Exactly-Once Resume

Jesun Ahmad Ushno
preprint en

Abstract

Agent frameworks promise durable, resumable agents, and the promise is rarely tested against a plain hand-written loop under matched controls. This study pre-registered the test. Two arms run the same 12 tasks with the same deterministic scripted policy, the same tools, the same step executor, and the same budgets, paired by seed: one arm sequences the work with LangGraph 1.2.20 and its SQLite checkpointer, the other with a hand-written loop over its own SQLite event log. The scripted policy is a declared scope limit: the study measures the orchestration layer only (sequencing, durability, recovery, re-execution, cost, code surface), not model quality. On task success the arms are equivalent with evidence: overall pass^1 is 0.979 in both arms (94 of 96 runs each), the mean paired per-task difference is 0.000 with a 90 percent confidence interval of [0.000, 0.000], inside the registered equivalence margin of plus or minus 0.05 (H1: Supported). Consistency (pass^k, k = 1 to 8) is identical in the two arms at every k. On cost, policy invocations are identical (median 5.0 per run in both arms, ratio 1.000 against a registered bound of 1.05) and LangGraph's median wall time per run is lower in these implementations (0.293 seconds against 0.980, ratio 0.299 against a bound of 1.10), a comparison the paper flags as implementation-sensitive because the native arm commits state synchronously after every step (H2: Supported). On recovery after a SIGKILL (48 fault runs per arm, two kill classes), both arms pass every run with every intended side effect applied exactly once and no duplicates: the exactly-once resume rate is 1.000 in both arms, so the registered tie form holds and the orchestrator adds nothing measurable on recovery in this design (H3: Supported, tie form). The difference is in what recovery costs: in the step-boundary kill class, LangGraph recomputed exactly one already-finished step in all 24 runs (its durable checkpoint lags the last finished step), while the native event log recomputed none in all 24; LangGraph also needed less arm-specific code (81 against 96 code lines) and a durable state store about 2.25 times larger (median 36,864 against 16,384 bytes per run). Exactly-once came from the idempotency discipline both arms applied, not from the orchestrator. The debugging axis (H4) is reported as Not yet measured, for the reason registered in advance: the arms persist traces in incommensurable units, so the registered record-count procedure would measure storage format, not diagnosis effort.

Zenodo (CERN European Organization for Nuclear Research)
Scientific Computing and Data Management
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon