A deep reinforcement learning algorithm with heterogeneous graph and hierarchical attention mechanism for dynamic flexible job shop scheduling problem

Dynamic flexible job shop scheduling (DFJSS) is widely encountered in manufacturing systems with dynamic events such as order changes and random new job insertions. Although exact, heuristic, and meta-heuristic methods have been widely applied to scheduling problems, real-time adaptive decision-making remains challenging in DFJSS because newly inserted jobs change the current shop-floor state and increase the complexity of machine assignment and operation sequencing. To address this issue, a deep reinforcement learning framework with heterogeneous graph and hierarchical attention mechanism (DRL-HGHAM) is proposed in this paper. In the proposed framework, jobs, operations, and machines are represented as different types of nodes in a heterogeneous graph, and their typed relationships are used to describe precedence constraints, machine eligibility, and resource competition. A hierarchical attention mechanism is then adopted to extract decision-relevant local and semantic information from the heterogeneous graph. Based on the learned state embedding, the reinforcement learning agent selects a suitable scheduling rule from five designed candidate rules to generate dispatching decisions. Experimental results on DFJSS instances with random new job insertions show that DRL-HGHAM achieves better tardiness performance and maintains good scalability compared with the tested baseline methods.