Adaptive Confidence-aware Preference-based Reinforcement Learning with Noisy Feedback

As AI systems become more integrated into society, ensuring they align with human values and exhibit responsible behavior is crucial. Preference-based Reinforcement Learning (PbRL), which learns directly from human feedback, offers a promising approach to achieving this alignment. However, PbRL faces significant challenges related to data quality, like inconsistent feedback, especially when crowdsourcing is used to collect feedback. To address these issues, we propose a method to enhance the robustness of PbRL algorithms with noisy preference data. Our approach provides a standardized framework for handling noisy feedback data by incorporating confidence analysis, temporal smoothing, and an adaptive penalty mechanism to reduce the impact of noisy data and improve learning stability. These techniques work synergistically to ensure the algorithm's ability to mitigate the negative effects of data imperfections and maintain a stable learning trajectory. We demonstrate the effectiveness of our approach, CA-PbRL, in the Meta-World environment, showing that it can recover the performance of existing PbRL algorithms under various noise conditions, thereby facilitating their application in real-world environments.