Replication Instability in Experience Replay for Multi-Agent LLM Pipelines

Abstract Experience replay, the re-injection of highly rated past outputs as in-context exemplars, gives large language model (LLM) agents memory without parameter updates. In a multi-agent system such a store is the organization's memory, and retrieval a coordination mechanism. Its effect is typically measured in a single stochastic run, which this paper shows cannot support the claim drawn from it. In a five-agent LLM pipeline, after repairing three coordination failure modes that had prevented the loop from closing, fifteen controlled comparisons across two models produced no replay effect that reproduces. Significance did not survive replication even where the sign did. Single-run evaluations produced large, significant effects of opposite sign on the two models (Δ=-0.172, p = 0.041 and Δ=+0.147, p = 0.040), and neither survived replication. A pool-size comparison that looked decisive in single runs did not survive either. Ten independent runs of one identical configuration, five per model, spanned a range of 0.138 and disagreed about significance and, on one model, about the sign. What relevance-matched retrieval can return is bounded by the resolution of its key rather than by the size of the store, and we prove the bound for key-dominated scoring; the parameter the literature reports is not the one that governs the outcome. An objective task with a deterministic answer key reproduced the instability. We conclude with a seven-point protocol to make such evaluations informative rather than merely confident. Code, data, and the full call log are available at https://github.com/wjh55224545/replication-instability-in-experience-replay.

Authors

Institutions

Publication Details

Journal
Research Square
Published
2026-11-12
DOI
https://doi.org/10.21203/rs.3.rs-11409743/v1
Primary Topic
Topic Modeling
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Replication Instability in Experience Replay for Multi-Agent LLM Pipelines

Ke Wu, shouqiang Liu, Jianhao Wang, Dong Liu
Research Square
Topic Modeling
preprint

Replication Instability in Experience Replay for Multi-Agent LLM Pipelines

Ke Wu, shouqiang Liu, Jianhao Wang, Dong Liu
preprint en

Abstract

Abstract Experience replay, the re-injection of highly rated past outputs as in-context exemplars, gives large language model (LLM) agents memory without parameter updates. In a multi-agent system such a store is the organization's memory, and retrieval a coordination mechanism. Its effect is typically measured in a single stochastic run, which this paper shows cannot support the claim drawn from it. In a five-agent LLM pipeline, after repairing three coordination failure modes that had prevented the loop from closing, fifteen controlled comparisons across two models produced no replay effect that reproduces. Significance did not survive replication even where the sign did. Single-run evaluations produced large, significant effects of opposite sign on the two models (Δ=-0.172, p = 0.041 and Δ=+0.147, p = 0.040), and neither survived replication. A pool-size comparison that looked decisive in single runs did not survive either. Ten independent runs of one identical configuration, five per model, spanned a range of 0.138 and disagreed about significance and, on one model, about the sign. What relevance-matched retrieval can return is bounded by the resolution of its key rather than by the size of the store, and we prove the bound for key-dominated scoring; the parameter the literature reports is not the one that governs the outcome. An objective task with a deterministic answer key reproduced the instability. We conclude with a seven-point protocol to make such evaluations informative rather than merely confident. Code, data, and the full call log are available at https://github.com/wjh55224545/replication-instability-in-experience-replay.

Research Square
South China Normal University (CN), South China Business College Guangdong University of Foreign Studies (CN)
Topic Modeling
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon