Preference-Based Multi-Objective Reinforcement Learning with Explicit Reward Modeling

Multi-objective reinforcement learning (MORL) often faces challenges in defining a single reward that captures complex real-world objectives. In this paper, we introduce Preference-based Multi-Objective Reinforcement Learning (Pb-MORL), which addresses this challenge by utilizing human preferences, as a more intuitive and flexible alternative to reward functions. Our key contribution is the development of an explicit multi-objective reward model, which aligns with preference data through the Bradley-Terry model and cross-entropy optimization. This model serves as a bridge between qualitative preferences and the quantitative needs of MORL, facilitating policy optimization without predefined rewards. Extensive experiments demonstrate that Pb-MORL achieves comparable performance to state-of-the-art MORL methods, showcasing its potential for broader real-world applications.