Task-Oriented Language Grounding for Robot via Learning Object Mask

For robots to follow human instructions, it is essential to associate the rich semantic information in human vocabulary, e.g. "pick up the red block" to their sensory observations and actions. This brings up a notably difficult challenge for robots: while current robot learning approaches allow robots to learn different behaviours from observational learning, it is impractical for robots to have experiences that cover all semantic information when faced with variable language instructions. We would enable the robot to perceive and manipulate task objectives and language instructions, even if it has never seen them before. In this paper, we propose a modular system and a FiLM-based language-visual object mask model (LOVM) to extract target masks of task objects from language instructions and images, and execute robotic manipulation strategies from the current image, the instructions and the extracted object information. we evaluate the generalization ability of LOVM for unknown language instructions and task goals using three test sets. Results demonstrate that our approach is effective for a wide range of language instructions, and achieves an average success rate of 92%.