Rethinking Urban Water Network Design: A Reinforcement Learning Framework for Long-Term Flexible Planning

Abstract Effectively planning the design of water distribution networks (WDNs) for their whole lifecycle is a complex task for water utilities due to the dynamic nature of WDNs, their long planning horizons, and the deep uncertainty that characterises key design parameters such as future water demand and population growth. Existing flexible design methods, which attempt to address these challenges, rely on static heuristic approaches and predefined decision pathways, requiring re-optimisation whenever new information becomes available. As additional scenarios are introduced, these methods also suffer from exponential increase in complexity, limiting their ability to adapt to emerging information and efficiently explore a wide range of future possibilities. In this work, we introduce a deep reinforcement learning (DRL) framework for the flexible, long-term design of WDNs. By formulating the least-cost staged design problem as a Markov Decision Process and training an agent using Proximal Policy Optimisation, our approach learns cost-effective, sequential interventions across multiple construction stages and future scenarios without relying on predefined decision trees. We evaluate our method on a modified New York Tunnels benchmark across three design tasks, ranging from static single-stage to flexible multi-stage design. Our results show that the DRL agent performs comparably to state-of-the-art heuristics for static and staged deterministic tasks. In the flexible design task, it autonomously devised adaptive strategies, clustered similar scenarios, and maintained high sample efficiency as the number of stages and scenarios increased. These findings highlight DRL as a promising alternative for the lifecycle design of WDNs, establishing a new paradigm for long-term water network planning under deep uncertainty.