An efficient deep reinforcement method for smart particle navigation in complex flows

Abstract As a quintessential example of soft matter, smart microswimmers bridge the gap between soft matter physics and functional robotics. The development of autonomous navigation of smart microswimmers in complex fluid environments is thus vital, addressing core challenges in the development of robotics in targeted drug delivery and precision surgery. Reinforcement learning is rapidly emerging as an effective solution for such challenges. Traditional deep Q-network (DQN) method often exhibits the limitations of insufficient exploration and low learning and sampling efficiency in complex fluid environments. To address these limitations, we present an efficient deep Q-learning-based approach, which incorporates a novel exploration strategy and an experience sampling strategy into the classic DQN method. The proposed approach enhances exploration through a learned network that generates state-dependent weights and improves sampling efficiency through the use of state-experience clustering in experience replay. We apply the proposed method to three particle navigation tasks in complex fluid environments and show that the proposed method outperforms many existing DQN-variants. The proposed approach enables the efficient calculation of optimal strategies, serving as an effective solver for intelligent navigation challenges across various physics and engineering scenarios.