Human-in-the-Loop Reinforcement Learning to Track Excavation Paths for a Hydraulic Excavator in a Short Period of Time

The construction industry faces a severe shortage of skilled operators. One promising approach to mitigate this problem is to automate construction machinery, particularly hydraulic excavators, which have a wide range of applications. However, it is challenging to automate hydraulic excavators due to the difficulty of modeling their nonlinear dynamics and soil interactions, which makes difficult to design a controller that tracks excavation paths precisely and quickly. Reinforcement learning (RL) is promising solution because it can learn to track excavation paths in a purely data-driven way, but simply applying RL may lead to an impractically long data collection time due to its low data efficiency. Moreover, since controlling the hydraulic excavator is a safety-critical task, the safety of RL is also a considerable challenge. In light of this background, we propose a path-tracking controller based on a human-in-the-loop RL system. While some human intervention is required, we can collect data safely and efficiently. Our experimental results demonstrate that the path-tracking controller assisted by a skilled human operator learned to excavate gravelly, sandy, and cobbly soils in 12, 18, and 21 minutes of data collection, respectively, and dispensed with human assistance afterward. Moreover, the results of safety analysis experiments show that the trained controller succeeded in performing 128 successive excavations (38 minutes) without any safety violation or human assistance.