Reward shaping based D3QN algorithm for solving capacitated lot-sizing problem with stochastic demand

Abstract In this paper, we study the capacitated lot-sizing problem (CLSP) in a single-product system with stochastic demand, aiming to minimize the total production cost, which includes production, inventory, machine setup, and stockout costs. We focus on the design and application of deep reinforcement learning (DRL) algorithms, proposing a reward shaping-based D3QN algorithm. First, we provide a detailed problem description and formulate it as a reinforcement learning model using a Markov decision process. Then, we adopt the aggregate modified base-stock heuristic (AMBS) algorithm as the teacher strategy and apply Potential Function-based Reward Shaping for policy transfer, guiding the agent to mimic the teacher’s behavior. Experimental results show that the RS-D3QN algorithm significantly outperforms other algorithms, with an average total cost 1.9% lower than the D3QN algorithm, 17.9% lower than the EOQ method, and 6.15% lower than the AMBS method. These results demonstrate that the RS-D3QN algorithm successfully combines the global optimization power of DRL with heuristic knowledge, addressing the challenges of low learning efficiency and slow convergence in complex production environments.