Enhancing Long-Term Decision Making in Trajectory Planning: Deep Reinforcement Learning With Knowledge Transfer and Online Demonstrations

Abstract Deep reinforcement learning is now widely applied in motion planning problems for autonomous systems due to its model-free nature and its ability to solve complex control problems through trial-and-error. However, the success of deep reinforcement learning depends heavily on the exploration policy and the design of the reward function. This dependence makes it challenging to solve long-range planning problems and requires careful reward function design to avoid the sparse reward problem. In this paper, we propose a dual-level off-policy deep reinforcement learning framework that combines high-level policy learning and low-level integration of a controller. The high-level policy enables the agent to make long-term decisions with a flexible horizon. Using a high-level policy also mitigates the sparse reward problem in long-range planning tasks. By storing only high-level actions and transitions in the experience buffer, the agent can efficiently learn in long-range trajectory planning tasks. The proposed parallel architecture of the neural network enables the utilization of high-level domain knowledge transfer. Online demonstrations can be integrated during training using global planning algorithms to enhance the quality of experiences. Experimental results show that transferring high-level geometry knowledge and applying online error correction can significantly enhance the agent’s performance in long-range trajectory planning tasks.