Demonstration-guided Reinforcement Learning for Optimizing Outbound Container Stacking Problem

Outbound container stacking is an important operation in ports, significantly impacting later retrieval for vessel loading. Stacking refers to designing an optimal strategy for placing arriving containers in stacks with the aim of minimizing the retrieval time. Out of stability and safety concerns, heavier containers are often placed at the bottom of the vessels and thus should be retrieved earlier from the yard. Consequently, it requires prioritizing heavier containers on the top of lighter ones when stacking. The challenge for the problem lies in the undetermined sequence of arriving containers, making it difficult to achieve this desired layout. Dynamic programming is the most predominant approach for addressing the stacking problem in the literature. However, it bears the curse of dimensionality when incorporating more state-related information. To address this challenge, we employ the Dueling Double Deep Q Network (D3QN) algorithm, a kind of reinforcement learning approach. It can leverage all the state features and eliminate redundant information, enabling more informed and efficient decisions. Additionally, we find that some types of relocations can be limited within some certain stacks by adopting the categorizing policy. We incorporate this idea as an expertise demonstration into the D3QN algorithm to further facilitate the training process. Finally, we validate the effectiveness of the proposed method based on the benchmark data, which shows the proposed demonstration-guided D3QN algorithm outperforms the existing methods for 90% of tested instances.