End-to-End Model for Vision-Language Navigation Based on Pre-Trained Model

Aiming at the problem that embodied intelligent mobile robots have difficulty in accurately navigating in unseen environments according to language instructions, this paper proposes an end-to-end model suitable for visual language navigation. The contextual representation of natural language instructions is extracted by the pre-trained BERT model, and then fed into the model's action predictor, which consists of a global predictor and a local predictor. The global predictor obtains high-level information of the scene through the topological graph to make global action planning. The local predictor only focuses on the visual information around the current position to make local action planning. The predictors are all cross-modal information fused through a transformer-based approach. Finally, the output of both predictors is dynamically fused into actions suitable for the robot. The proposed model is evaluated in the Matterport3D simulator to verify its effectiveness and superiority. Experiment results show that the navigation success rate is improved and mean navigation error is decreased compared with previous work, which can effectively improve the generalization ability of robot navigation in unseen environments.