Toward safe and efficient shield tunneling: counterfactual reinforcement learning based multi-subsystem collaborative optimization

Abstract Shield tunneling must satisfy safety requirements while maintaining efficiency, but the dynamic interactions among subsystems and continuously evolving ground conditions make manual coordination unreliable. To address this challenge, this study proposes the Counterfactual Reinforcement Learning-Based Shield Multi-Subsystem Collaborative Optimization (CRL-SMSCO), a multi-agent reinforcement learning framework with a centralized critic and decentralized actors. CRL-SMSCO performs counterfactual credit assignment to quantify each subsystem’s marginal contribution to global tunneling outcomes and jointly optimizes subsystem parameters. Safety is enforced by a safety-oriented action masking unit that restricts the feasible action space in real time, together with a hierarchical reward that prioritizes safety over efficiency. These features enable CRL-SMSCO to achieve interpretable subsystem coordination and rigorous safety enforcement. In the Nanjing Metro case study, experiments show that CRL-SMSCO improves average training reward by 3.9% over multi-agent deep deterministic policy gradient (MADDPG) and by 3.5% over multi-agent proximal policy optimization (MAPPO), while also yielding lower variance and higher minimum rewards on the held-out test segment. In the engineering application, relative to experience-based manual operation under similar geology, CRL-SMSCO increases average tunneling speed by 13.2%, reduces equipment loads over 1.7%, and decreases ground deformation by 82%, with all indicators maintained within permissible limits. These results demonstrate that CRL-SMSCO provides significant practical value in shield tunneling, offering an effective framework for managing other safety-critical coupled systems.