Physical State Exploration for Reinforcement Learning from Scratch

Although reinforcement learning (RL) algorithms have demonstrated impressive capabilities in simulation, their transition into the real-world often reveals a performance gap. A key challenge of real-world deployment is ensuring robustness to complex or unmodeled physical phenomena, such as anisotropic friction during locomotion. This discrepancy between simulated and real-world performance underscores the need for hardware benchmarks to rigorously evaluate and improve RL algorithms for physical deployment. In this work, we present NoodleBot—a low-cost, untethered three-link swimmer robot—as a hardware benchmark for RL algorithms. This benchmark is intended to complement embodied learning approaches, where simulations provide confidence that algorithms are able to learn from scratch in challenging environments. We demonstrate three algorithms learning on the platform and compare the results to learning with a simulated swimmer, highlighting the importance of effective state exploration to agent performance. We also show the ability of one algorithm to learn in single-shot hardware deployments. Design, firmware, and software are available open source at https://github.com/MurpheyLab/NoodleBot.