Learning Cross-modal Knowledge Reasoning and Heuristic-prompt for Visual-language Navigation

Visual language navigation is an exciting and challenging multi-modal task. Most existing research focuses on the fusion of visual features and semantic space, which ignoring the importance of local highlight features and semantic knowledge alignment in images for agent navigation. Therefore, this paper proposes a novel visual language model combining Knowledge-augmented Reasoning and Soft-Prompt (KRSP) learning. First, we perform fine-grained processing of local regions in the image and to map context image features and text knowledge to the same common sub-space. We focus on regional knowledge to increase the model reasoning ability. Next, soft-prompt learning aligns keywords and sub-visual information in instruction features to solve the path mismatch problem in coarse-grained instructions. We use a large-scale pre-training model CoCoOp to collect highly matched soft action prompts into a unified instruction set. Finally, we propose a general cross-modal feature alignment loss function. The potential semantic correlation between sub-visual information and instruction space is closer through the penalty mechanism of the alignment function. This paper verifies the method effectiveness on the R2R and REVERIE datasets, and the experimental results show that KRSP achieves state-of-the-art performance. Among them, the KRSP of SPL evaluation metric increased by 4.5% in unseen scenarios.