NavGemini: a multi-modal LLM agent for vision-and-language navigation

Abstract Large language models (LLMs) such as the GPT series exhibit impressive reasoning and in-context learning capabilities due to the substantial amount of data and computational resources involved in LLM training. Some previous studies have applied the LLMs to vision-and-language navigation (VLN) in order to create navigation agents that are entirely LLM-based, operating within a zero-shot setting, aiming to reveal and utilize LLMs’ reasoning and planning capability for VLN tasks. However, these methods employ text-based LLMs for navigation agents, generating a text description of environmental observations during navigation. Other smaller LLMs are used for image-to-text translation, resulting in a gap between image-to-text translation and environmental navigation. Moreover, the high cost of advanced LLMs such as GPT-4 also hinders the application of LLM-based navigation agents. This is particularly the case given the increasing length of the context, which includes navigation history and the numerous visual images generated during navigation. In this paper, we propose NavGemini, a navigation system based entirely on the newly developed multi-modal LLM, Gemini-Pro-Vision. Our aim is to study and utilize the visual-spatial and multi-modal capabilities of LLMs in VLN tasks, while mitigating the challenges posed by token limits when LLMs process large amounts of image-based data and historical information, and when the available multi-modal LLMs perform relatively poorly. Our proposed NavGemini, with its elaborate prompts, successfully outperforms previous methods by 5.7% in terms of success rate, even when using inferior LLMs. This demonstrates the strong ability and potential of multi-modal LLM-based agents in VLN tasks.