Towards ethically aligned agents with multi-objective reinforcement learning

Abstract Reinforcement learning (RL) has been increasingly applied to solve a large range of real-world problems, from autonomous vehicles to healthcare. Multi-objective reinforcement learning (MORL) extends RL by enabling the balance of multiple, sometimes conflicting objectives, making it particularly suited to model such complex applications. As MORL gains traction and its applications expand, an increasing number of MORL-based agents are expected to be deployed across our societies. This however raises ethical questions, as these autonomous agents can have a large impact on humans. Although multi-objective RL has shown promise as a framework to develop human-aligned artificial intelligence, its potential is still largely unexplored and no survey has yet addressed this gap. Hence, we provide a detailed review of multi-objective reinforcement learning through an ethical perspective. We propose five key axes to consider for developing ethically aligned agents, and use them to highlight existing works, gaps in the literature, and potential areas for future research. We further examine the difficulty of benchmarking ethical alignment in realistic environments, and illustrate it through a case study applying representative methods to a smart-grid simulator. Ultimately, we hope this work can be useful to both machine ethics and multi-objective reinforcement learning researchers, as a step towards bridging the current divide between algorithm-focused and ethics-focused advancements in MORL.