Cosine Similarity Based Adaptive Implicit Q-Learning for Offline Reinforcement Learning

Offline Reinforcement Learning (RL) methods constrain the policy to align with the behaviour policy, mitigating extrapolation errors caused by out-of-distribution (OOD) actions. Implicit Q-Learning (IQL), a popular offline RL algorithm, leverages expectile regression and introduces an in-sample learning paradigm that enhances the policy evaluation stage without querying OOD actions. However, the crucial parameter <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$T$</tex> for expectile regression in IQL is fixed, limiting both its performance and flexibility across diverse datasets. In this paper, we propose Cos-IQL, an improved IQL approach based on cosine similarity, which optimizes the policy evaluation function by measuring the cosine similarity between the policy and the behaviour policy. Cos-IQL is essentially a multi-step offline RL algorithm but retains the advantages of in-sample learning, thus avoiding the risks of OOD actions. In addition, Cos-IQL can adaptively adjust parameter <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$T$</tex> without elaborate fine-tuning. We evaluate Cos-IQL on D4RL benchmark datasets and compare its performance against recent competitive offline RL algorithms. Experimental results show that Cos-IQL achieves state-of-the-art performance.