Enhancing Path Planning with Self-Imitation and Intrinsic Curiosity in Soft Actor–Critic

The integration of unmanned aerial vehicles (UAVs) into low-altitude urban airspace for parcel delivery and last-mile logistics is emerging as a key component of future urban air mobility, yet its operational viability depends on autonomous path planning that is simultaneously safe, reliable, and efficient under dense obstacles and limited sensing. Deep reinforcement learning offers a promising planning paradigm, but its deployment in this transport setting is severely constrained by sparse task-level success signals, low sample efficiency, and a poorly regulated exploration–exploitation trade-off. To address these issues, this paper proposes an improved Soft Actor–Critic (SAC) algorithm named SIL-CAT-SAC, which integrates self-imitation learning (SIL) and a curiosity-adaptive temperature mechanism. Specifically, a curiosity-augmented target entropy adjustment (CAT) module based on enhanced random network distillation is designed to dynamically regulate the exploration intensity according to state novelty, achieving instance-level adaptive temperature control; a linearly annealed curiosity coefficient further prevents intrinsic rewards from overriding the delivery objective near convergence. A self-imitation learning auxiliary loss combined with prioritized experience replay is introduced to selectively amplify genuine successful trajectories, eliminating the interference of pseudo-success samples from hindsight experience replay and markedly improving the utilization of rare positive samples. In addition, a continuous reward function incorporating distance increment, potential shaping, and obstacle warning terms is constructed to alleviate gradient vanishing under sparse success feedback. Comprehensive experiments are conducted in a 35×35 grid-based urban path planning environment across three obstacle densities, against two state-of-the-art SAC variants, REDQ and M-SAC. All reported metrics are averaged over three independent training seeds; where the dispersion across seeds is meaningful, the cross-seed standard deviation is reported together with the mean. The results show that, in a frozen policy evaluation against the fixed goal, SIL-CAT-SAC attains the highest success rate at each of the three obstacle densities, and the exact Friedman test rejects the null hypothesis of equal performance at the two higher densities. It reaches its operating success level in fewer training episodes than either baseline, and it exhibits the largest average robustness index among the three algorithms. A factorial ablation at the lowest density, where none of the four configurations reaches the success ceiling, separates the component combinations: removing either component lowers the success rate and delays convergence, while the joint effect is no larger than the sum of the individual effects, so both components contribute and no positive CAT-SIL interaction is claimed. The study is conducted in a two-dimensional grid-based abstraction with static obstacles; three-dimensional flight dynamics, dynamic obstacles, wind disturbance, sensing uncertainty, and communication constraints are outside the scope of the present evaluation, and experiments extending the framework to three-dimensional environments are underway and will be reported in a follow-up paper. Within this simulated two-dimensional grid-based setting, the results support a success rate and convergence efficiency advantage for the proposed method at every density, as well as a cross-density robustness advantage at the intermediate density; the advantage is attributed to the two components acting in combination rather than to a demonstrated positive interaction between them. Validation in three-dimensional airspace with dynamic obstacles, sensor uncertainty, and realistic vehicle dynamics remains necessary before operational deployment can be claimed.