Diffusion Actor with Behavior Critic Guidance Algorithm for Offline Reinforcement Learning

To address the multimodal nature of offline dataset distributions and the overestimation problem associated with out- of-distribution (OOD) actions, this paper introduces the diffusion actor with behavior critic guidance (DABCG) algorithm. The proposed method first employs diffusion models to model the target policy, incorporating a Q-value guidance term into the actor loss to enhance the algorithm’s expressiveness while achieving performance that surpasses the behavior policy. The introduction of Q-value guidance ensures that the learned actions are directed toward higher-value regions of the action space, thus facilitating enhanced policy optimization. Furthermore, by integrating behavior critic guidance into the policy evaluation process, the algorithm effectively constrains the overestimation of OOD actions, ensuring robust performance. Experimental results on the D4RL benchmark dataset provide strong empirical evidence supporting the efficacy of the DABCG algorithm.