Reward Shaping for RL Based on Human Multi-Attribute Scaling Preferences Feedback

This study focuses on offline Preference-based Re-inforcement Learning (PbRL), a variant of traditional rein-forcement learning (RL). Reward is an important component of RL and designing a reward function that fully aligns with human intentions is challenging. To address this, PbRL provides a framework where human teachers select their preferred one from pairs of segments, allowing the agent to learn a reward from this feedback. However, existing methods have limitations such as unclear preference strengths and considerations. We propose Scaling Preference (SP) and Multi-Attribute Scaling Preference (MA -SP) feedback schemes for offline environments. These methods allow human teachers to provide more detailed and clear guidance by considering different attributes of a task. We tested our methods on 12 tasks across 3 environments in the D4RL and Robomimic benchmarks. The experiments show that our methods are highly competitive compared to baselines.