Cooperative Relaying for Connected Construction Equipment Networks With Hybrid Hierarchical Proximal Policy Optimization

The communication network in a tunnel construction site facilitates real-time data exchange, and serves as a backbone for successfully executing construction projects. However, the long and closed spaces, irregular surfaces, and variable topology as tunnel excavation impose rigorous limitations on signal propagation, communication quality and coverage. To alleviate the realistic issues, we introduce a holistic three-phase cooperative relay scheme based on 5G New Radio (NR) vehicle-to-everything (V2X) architecture, which can extend the communication range and enhance network throughput. We theoretically derive the outage probability of the entire cooperative relaying process from source to destination, and quantify the impact of relaying on construction workflow with relay cost. To minimize the outage probability and relay cost, we formulate a cooperative relay strategies optimization problem and transform the solving procedure into a Markov decision process (MDP). We design a hybrid hierarchical proximal policy optimization (HH-PPO) reinforcement learning method to solve the MDP, which consists of two discrete actor networks, two continuous actor networks, and two critic networks. The hybrid structure enables HH-PPO to tackle the mixed action space, and the hierarchical structure enables adaptive and contextual actions generation by integrating the discrete network outputs into the continuous actor network. Simulation results validate the effectiveness of the HH-PPO algorithm with faster convergence speed, and show superior performance in terms of lower, stable outage probability and relay cost satisfaction compared with another benchmark.