Reinforcement Learning for Sequential Truck–Drone Routing with Fixed Vehicle Assignments: A Comparative Evaluation of Heuristic, Metaheuristic, and Learning-Based Approaches

This study evaluates a reinforcement learning policy for a deterministic truck–drone routing problem with fixed customer-to-vehicle assignments. Truck travel and drone sorties occur sequentially, and each sortie returns to its launch location while the truck waits. A Maskable Proximal Policy Optimization (PPO) policy, trained using behavior cloning and a curriculum of increasing instance sizes, is compared with Random Valid, Nearest Neighbor (NN), Drone Priority, and Variable Neighborhood Search (VNS). The evaluation comprises 100 matched test instances at each of four sizes, with 10, 20, 30, and 40 customers. PPO reduces the mean mission makespan relative to NN by 2.90% to 7.52%, while its mean makespan remains 12.50% to 17.28% above that of VNS. Paired bootstrap intervals support these mean differences conditional on the evaluated policies. All methods complete every test instance without invalid actions. Additional experiments include three independent training seeds, component ablations, a fixed-checkpoint evaluation at unseen customer counts, and controlled online timing. Improvements over NN persist across the three full-framework seeds, while cloning alone performs similarly to the full framework. An analytical decomposition separates open truck routing from independent drone launch assignments. A heuristic exploiting this structure achieves mean per-instance optimality gaps of 0.85% to 2.35%, compared with 14.19% to 32.31% for the original PPO checkpoints, against reference solutions certified for all 400 instances. The controlled timing evaluation does not establish a solution quality–runtime advantage for PPO over the decomposition heuristic. The findings support a comparative assessment of learning under the restricted operating model, rather than superiority over structure-exploiting optimization.