A Feedback-based Decision-Making Mechanism for Actor-Critic Deep Reinforcement Learning

Deep reinforcement learning (DRL) has achieved remarkable success in solving sequential decision-making problems across various domains. However, a critical challenge is sample inefficiency, especially in real-world environments with high-dimensional solution spaces due to continuous state and action spaces. Although off-policy actor-critic algorithms have been proposed to mitigate this issue, the gains in sample efficiency remain limited, as decision-making in these algorithms relies solely on the policy function that might not always yield optimal actions. To bridge the gap, we design a novel feedback-based decision-making mechanism (FADA) that incorporates a feedback mechanism into the actor-critic framework to enhance decision-making robustness. Specifically, FADA utilizes feedback from the value function (critic) to calibrate the decisions produced by the policy function (actor). More concretely, FADA comprises four integrated modules: a decision-space expansion module (DEM) to produce a pool of candidate actions, a critic-guided evaluation module (CGEM) that estimates the efficacy of the candidate actions, an adaptive selection module (ASM) that adaptively selects a set of elite actions based on the estimated efficacy and samples the final action, and an iterative refinement module (IRM) that improves the quality of elite actions. We evaluate our approach on multiple tasks in the DeepMind Control Suite, and the experimental results demonstrate a significant improvement in sample efficiency.