USV-3.0: Cognitive maritime navigation through vision-language models, Human-in-the-Loop learning, and spatio-temporal memory

• A cognitive navigation framework (USV-3.0) integrating VLMs, HITL and memory. • A spatio-temporal navigation memory enabling reuse of learned visuomotor behaviors. • One-shot learning of maritime navigation from 23–61 human demonstrations. • Confidence-based human-in-the-loop mechanism for safe autonomous operation. • High-fidelity validation in realistic maritime scenarios with 98.2% autonomy. Current Unmanned Surface Vessel (USV) systems follow the USV-1.0 paradigm (traditional modular perception-planning-control stacks) or more recently the USV-2.0 approach (end-to-end neural policies requiring large training datasets). In contrast to being effective in constrained settings, both paradigms fail in leveraging accumulated human nautical expertise and struggle with data efficiency and interpretability. In this paper we propose a cognitive maritime navigation paradigm that integrates Vision-Language Models (VLMs), Human-in-the-Loop (HITL) learning, and Spatio-Temporal Navigation Memory (STNM) to enable autonomous surface vessels to learn and reuse visuomotor behavior from human demonstrations, which we nominate as USV-3.0. This novel approach integrates three components into a unified cognitive framework: (1) language-conditioned maritime vision-language models (VLMs) that interpret 360 ∘ visual scenes and ground perception in the intent conveyed by natural-language mission commands; (2) a confidence-based HITL mechanism that triggers operator intervention when system confidence drops below 90%, storing validated human decisions together with their linguistic and visual context; and (3) an STNM system that records trajectories, visual embeddings, and textual reasoning, enabling hybrid GPS- and CLIP-based retrieval of past behaviors conditioned on mission semantics. We validate USV-3.0 in VisualSimBoat using four real-world navigation scenarios that represent realistic conditions. Considering trained configurations (normal visibility), the system achieves 100% mission success, 98.2% autonomous functioning, and memory retrieval precision within 8m, learning new routes from 23 to 61 HITL episodes-orders of magnitude fewer than typically required by USV-2.0 approaches. Zero-shot transfer experiments under adverse visibility conditions reveal limitations in decision execution, motivating further investigation into safety mechanisms for unseen environments. These results position language-conditioned cognitive navigation as a data-efficient alternative to existing USV paradigms, while clearly delineating its current boundaries and future research directions.