Efficient Frontier Selection via Reinforcement Learning for Exploring Unstructured Environments with Minimal Sensing

In recent years, reinforcement learning (RL) has been applied to frontier-based exploration to enhance a robot’s decision-making policy and improve exploration performance. In this work, we address this scenario with the aim of pushing forward the finding of the optimal frontier selection policy in unknown, unstructured environments, with RL deployed for a minimal sensing drone setup. We propose a novel policy architecture, featuring an attention module that uses the global map features captured by a convolutional neural network together with local frontier features in the form of scalar values, trained end-to-end with a scoring network using the Proximal Policy Optimization algorithm over a 2D randomized unstructured environment. Our approach demonstrates improved exploration efficiency in the evaluated scenarios, as it surpasses purely heuristic-based frontier selection strategies used as baselines for other RL methods, achieving shorter paths than the Nearest Frontier, the Hybrid Approach, and the TARE local horizon, as well as one-shot sim-to-real policy deployment.