Teaching Text Agents to Learn Sequential Decision Making from Failure

Text-based reinforcement-learning agents improve their policies by interacting with their environments to collect more training data.However, these self-collected data inevitably contain intermediate failed actions caused by attempting physically infeasible behaviors and/or hallucinations.Directly learning a policy from such trajectories can reinforce incorrect behaviors and reduce task success rates.In this paper, we propose a failed action-aware objective that suppresses the negative impact of failed actions during training by assigning zero return based on textual feedback.Building on this objective, we introduce a perturbation method that leverages unsuccessful trajectories to construct new successful ones that share the same goal.This allows agents to benefit from diverse experiences without further interaction with the environment.Experiments in ALFWorld and ScienceWorld demonstrate that our method significantly outperforms strong baselines and generalizes across environments.Code is available at https://github.com/ riken-grp/text-agent.