One Successful Run Is Not an Exploit: Replaying LLM-Written Exploit Scripts with the Model Out of the Loop

Abstract Offensive-security benchmarks for large language model (LLM) agents score a run as successful when a flag string appears in its output; published systems commonly stop at the first success. One verified run leaves two questions open at once: whether the model will produce that result again, and whether the script itself still works. We separate the two by taking the model out of the loop. Across 14 web challenges, 280 agent runs yielded 196 winning scripts, and each script was replayed verbatim, R = 10 times, into a freshly started target container. Cold replay recovered the flag in 1,950 of 1,960 replays (99.49\%, task-clustered bootstrap 95\% confidence interval 98.37 to 100.00), classifying 195 artifacts as stable, none flaky, one dead at (R = 10, cold, reference environment). The single dead artifact is the informative case: it fails 0/10 alone and succeeds 10/10 when its trajectory is replayed in order, because it hard-codes a subscript valid only in a primed target process; one extra request flips it (Fisher exact test, p = 1.08e-05). Instability instead concentrates in generation: 12 of the 24 cells with at least one hit are flaky there. Replaying an artifact and re-running an agent measure different quantities, and single-run scoring reports neither.