How Much Is an Agent's History Worth? A Law of Evaluation Transfer Across Agent Versions, Tested in Pre-Registered Studies

Each new version of an AI agent is evaluated from scratch, although earlier agents’ outcomes on the same tasks are known. We organise this history as a graph of agents—nodes are agent versions with their task-level outcomes—and ask how many tasks it is worth. A closed-form law answers: testing m tasks with the graph is as precise as testing m/(1−ρ2 ) at random, where ρ² is the share of the new version’s task-level variance the graph explains (φ² with the predecessor, η² with the difficulty seen by earlier agents). We turn it into a method: the graph chooses the tasks, a cross-fitted estimator estimates with calibrated intervals, and the same tasks decide keep-or-replace. In eight studies, each pre-registered by a git tag, the predecessor law, given the true φ², matched the observed gain within 3.9% (median) over 219 real version transitions from five benchmarks, and the graph law, given the true η², within 7.7% over 90; with φ² estimated from 20 tested tasks the median per-transition error, over simulated draws, was 5.3% (exploratory), but a single evaluation erred by 20%. Choosing the tasks gives most of the gain: with the cross-fitted estimator, 20 graph-chosen tasks were worth 32–44 random ones in two confirmations and 90% intervals covered the truth 0.944–0.959 of the time; decisions taken on the same tasks erred 32–54% less often when the versions differ by at least 5 points. On 13 newer AppWorld agents, four of them multi-agent systems, the method beat random testing in 24 of 26 evaluations (×1.31). Agents sharing a scaffold or an agentic pattern fail on the same tasks, without making evaluation more efficient. When results arrive asynchronously and evaluation stops at a deadline, naive estimates overshoot by 6.3 points; a correction learned from older agents did not transfer. Of 43 pre-registered hypotheses, 35 were supported; all failures are reported.