Synergistic Q-Functions and Risk Weighted Actor-Critic for Offline Reinforcement Learning

Offline reinforcement learning (RL) algorithms, which rely solely on static datasets, face significant challenges: (1) Insufficient coverage of the offline dataset can lead to substantial errors when the agent selects actions outside the dataset for policy learning, causing a severe deviation between the training policy and the behavior policy. (2) The quality of offline data entirely dictates the training of offline RL policies, resulting in poor generalization ability in new environments. To address these issues, we propose a novel Synergistic Q-functions and Risk Weighted Algorithm (SQRWAC). This algorithm innovatively integrates strategies in both the critic and actor designs to tackle environmental incompleteness and data sparsity. The critic incorporates a risk factor in policy evaluation to prioritize actions with high impact on rewards, optimizing long-term returns. Simultaneously, by synergistically computing the target value function using the Q-function, the algorithm alleviates over-fitting and enhances decision-making robustness. Comparative experiments in the MuJoCo simulation environment demonstrate that SQRWAC outperforms existing baseline algorithms, showing superior performance and higher data utilization efficiency across various standard RL tasks.