Reinforcement Learning with Partially Defined Rewards and Human Feedback for Energy Efficiency Recommendations

Most Reinforcement Learning (RL) pipelines require the explicit definition of a reward function that closely matches the goal of an RL agent, which is not trivial in some use cases, e.g. when user satisfaction is part of the objective, such as in energy efficiency recommendations.Reinforcement Learning from Human Feedback (RLHF) is a recently proposed RL paradigm in which user feedback is used to train a reward model without the need for an explicit reward definition, and it first appeared in Large Language Models (LLMs).However, most RLHF approaches focus only on processing human feedback and do not consider other parameters, such as costs, which are crucial in domains like energy optimization.Overall, a gap in the literature exists regarding RLHF applications within building energy efficiency optimization, especially in terms of partially defined rewards since existing methods assume that the reward function is fully defined without considering user feedback data.In this work, an RLHF approach with partially defined rewards is proposed, offering energy efficiency recommendations that optimize energy costs while learning user behavior based on collected feedback instances.The proposed approach fits a reward model using a feedback collection mechanism based on Markov Decision Processes for limited human feedback cases.Experiments show that the proposed pipeline outperforms existing RL approaches with explicit rewards, in terms of energy costs and achieved rewards, while it does not need a fully defined reward function.