Bayesian‐Belief Direct Policy Search for Adaptive Water Supply Planning With Endogenous Learning

Abstract Climate change uncertainty challenges water supply planning, where long‐lived infrastructure must ensure reliable supply under evolving conditions. Adaptive planning addresses this by incrementally expanding infrastructure only as needed, reducing unnecessary investments. Direct Policy Search (DPS), a reinforcement learning approach, has been widely used to identify adaptive rules that specify when and how much to expand infrastructure based on system conditions. However, standard DPS assumes static climate uncertainty, overlooking the potential to update uncertainty as new information emerges, potentially leading to over‐ or under‐investment as the climate evolves. We introduce Bayesian‐Belief DPS, a novel framework that integrates Bayesian learning into DPS to account for learning about climate uncertainty in adaptive planning. We do this by expanding the DPS state space to include a belief state, representing evolving climate uncertainty. The belief state is updated using Gaussian process regression as new observations become available. For instance, uncertainty about end‐of‐century climate is greatest early on but declines over time as data accumulates. We apply Bayesian‐Belief DPS to a case study in Mombasa, Kenya, where decision rules optimize infrastructure development over a 100‐year horizon. We compare policy performance across diverse climate and infrastructure scenarios to assess when learning improves planning. Results suggest Bayesian‐Belief DPS enhances robustness and cost‐effectiveness, especially under nonlinear climates, where past trends do not linearly predict future change, and for long‐lived investments, where incorrect assumptions about future climate can lead to high regret. By endogenously modeling climate beliefs, Bayesian‐Belief DPS offers a scalable, generalizable framework for adaptive planning under deep uncertainty.