APRES: Adaptive Policy-Aware Prioritized Experience Sharing for Scalable Decentralized Multi-Agent Reinforcement Learning

Decentralized cooperative Multi-Agent Reinforcement Learning (MARL) faces sever non-stationarity and sample inefficiency, particularly in sparse-reward environments where informative experiences may be discovered by only a subset of agents. Experience sharing can improve exploration by allowing agents to reuse knowledge acquired by their peers; however, indiscriminate sharing may introduce redundant or conflicting information and become increasingly problematic as the number of agents grows. Existing approaches generally do not jointly consider (1) which experiences are useful to a receiver, (2) whom to learn from, and (3) When does sharing become more beneficial? To address this challenge, we present Adaptive Policy-aware Prioritized Experience Sharing (APRES), a lightweight and fully decentralized experience-sharing framework that addresses these three dimensions. First, APRES employs receiver-aware advantage prioritization, using the receiving agent's critic to identify shared experiences which are most informative relative to its current knowledge. Second, it introduces policy-aware weighting, based on clipped importance-sampling ratios to regulate the contribution of experiences generated by behaviorally different peers while limiting unstable gradient amplification. Third, an adaptive temporal decay schedule progressively reduces the contribution of shared experience as training advances, balancing early-stage collaborative exploration with late-stage policy specialization. Experiments on sparse-reward cooperative MARL benchmarks show that our approach achieves higher performance while training on less shared experiences, improves sample efficiency, and scales significantly better than state-of-the-art experience-sharing baselines.

Authors

Institutions

Publication Details

Journal
Research Square
Published
2026-11-18
DOI
https://doi.org/10.21203/rs.3.rs-11362220/v1
Primary Topic
Reinforcement Learning in Robotics
Type
preprint

Funders

Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

APRES: Adaptive Policy-Aware Prioritized Experience Sharing for Scalable Decentralized Multi-Agent Reinforcement Learning

Mehdy Roayaei, Arefeh Esmaili
1 links
Research Square
Reinforcement Learning in Robotics
preprint

APRES: Adaptive Policy-Aware Prioritized Experience Sharing for Scalable Decentralized Multi-Agent Reinforcement Learning

Mehdy Roayaei, Arefeh Esmaili
preprint en

Abstract

Decentralized cooperative Multi-Agent Reinforcement Learning (MARL) faces sever non-stationarity and sample inefficiency, particularly in sparse-reward environments where informative experiences may be discovered by only a subset of agents. Experience sharing can improve exploration by allowing agents to reuse knowledge acquired by their peers; however, indiscriminate sharing may introduce redundant or conflicting information and become increasingly problematic as the number of agents grows. Existing approaches generally do not jointly consider (1) which experiences are useful to a receiver, (2) whom to learn from, and (3) When does sharing become more beneficial? To address this challenge, we present Adaptive Policy-aware Prioritized Experience Sharing (APRES), a lightweight and fully decentralized experience-sharing framework that addresses these three dimensions. First, APRES employs receiver-aware advantage prioritization, using the receiving agent's critic to identify shared experiences which are most informative relative to its current knowledge. Second, it introduces policy-aware weighting, based on clipped importance-sampling ratios to regulate the contribution of experiences generated by behaviorally different peers while limiting unstable gradient amplification. Third, an adaptive temporal decay schedule progressively reduces the contribution of shared experience as training advances, balancing early-stage collaborative exploration with late-stage policy specialization. Experiments on sparse-reward cooperative MARL benchmarks show that our approach achieves higher performance while training on less shared experiences, improves sample efficiency, and scales significantly better than state-of-the-art experience-sharing baselines.

Research Square
Tarbiat Modares University (IR)
Iran National Science Foundation
Reinforcement Learning in Robotics
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon