Regulating Replay Concentration in Prioritized Experience Replay Through Normalized Replay Effective Support

This study investigates whether replay concentration in prioritized experience replay can be measured and regulated directly through normalized replay effective support, and how such regulation affects reinforcement-learning performance across tasks. Fixed-exponent prioritized replay is first examined in three low-dimensional control environments and five MinAtar games to characterize how realized replay support varies across tasks and training stages. An adaptive controller is then introduced to adjust the priority exponent so that replay support satisfies a prescribed floor. Replay concentration is evaluated using effective support together with entropy, probability-mass concentration, and sampling-diversity diagnostics, while learning performance is assessed using final return and time-normalized area under the evaluation curve. Experiments show that a fixed priority exponent does not correspond to a consistent replay-support state across tasks, whereas adaptive control can maintain a common support target with high recalibration accuracy and limited computational overhead. However, learning outcomes remain strongly task dependent. Clear improvements are observed in some MinAtar environments, particularly Seaquest, while other tasks show limited, inconclusive, or unfavorable outcomes relative to uniform replay and established prioritization baselines. These findings show that replay effective support is useful for characterizing and regulating sampling concentration, but accurate control of the replay distribution alone does not guarantee improved learning performance.