Episodic Future Thinking With Offline Reinforcement Learning for Autonomous Driving

The Internet of Vehicles (IoV) is a network that connects various transportation devices, where autonomous vehicles play a crucial role. These vehicles must be able to coexist and drive alongside human-driven vehicles. To ensure a safer IoV, it is essential to develop adaptive decision-making capabilities, even under the uncertainty posed by human drivers. This study aims to develop an autonomous vehicle with the capacity for foresighted decision-making in complex multi-agent interactions by integrating predictive abilities of future observations. The proposed autonomous vehicle is equipped with episodic future thinking (EFT) capabilities. Specifically, EFT uses a prediction network to simulate future observations and guide adaptive decision-making of actor-critic networks based on envisioned scenarios. We use an offline reinforcement learning paradigm to train both the prediction network and the EFT-based actor-critic networks. To assess the generalized performance of the proposed solution, we integrate the EFT module with three popular offline reinforcement learning algorithms. We run an extensive series of performance evaluations across three driving scenarios involving multi-agent interactions and use four datasets for offline training, including a real-world dataset. Simulation results demonstrate that the proposed solution with the EFT module improves average performance in most cases compared to the baseline without the EFT module. Additionally, we compare the proposed solution with three existing model-based reinforcement learning approaches, confirming its superiority. Finally, we analyze driving behavior in terms of agility, safety, and stability.