Vision-and-Language Navigation with Perspective Taking and Knowledge Graph

Vision-and-Language Navigation (VLN) is a significant natural navigation task in human-robot interaction environments, which requires a robot to navigate according to natural language instructions and visual information. In unknown environments, the VLN tasks will be confronted with great challenges due to the uncertainty of the unseen environments and the natural language instructions. Hence, it plays a crucial role for the robot to interact efficiently with humans and to have common sense knowledge for better performance in VLN tasks. In this paper, we propose an effective vision-and-language navigation approach with spatial perspective taking and knowledge graph. First, spatial perspective taking is introduced into the VLN tasks by considering the selection of reference frames to avoid the ambiguity of human natural language instructions, where the selection preference of reference frames is estimated in an actual conversational situation for the design of human-centered robots. Second, we propose and implement two VLN models, i.e., Graph Cross-Modal Reasoning Navigator (GCMRN) and Attention Graph Convolutional Network Cross-Modal Recognition Reasoning Navigator (AGCMR2N), using knowledge graph provided with prior knowledge in language and vision. Finally, several groups of experiments are implemented in a continuous 3D environment, and the results demonstrate the improved performance of the proposed approach.