A Prompt-Driven Vision-Language Framework for Deictic Interpretation in Human-Robot Handover

Recent advancements in Vision-Language Models (VLMs) have enabled robotic systems to leverage model-based understanding and reasoning over visual and linguistic inputs, offering a promising approach for interpreting user intent in human–robot interaction (HRI). In particular, deictic expressions commonly used in object handovers, such as “take this” and “give me that”, cannot be fully interpreted through language alone and require a comprehensive understanding of the speaker’s perspective and the environment. This study proposes a prompt-driven vision-language framework for deictic interpretation in human–robot handover. The system integrates a pre-trained VLM with a hierarchical prompt that decomposes reasoning into intent classification, spatio-temporal grounding, and output self-validation, enabling accurate identification of target objects and goal locations without model fine-tuning. Experimental results demonstrate 100% command interpretation accuracy across multiple interaction scenarios, including pick-and-place tasks, robot-to-human and human-to-robot handovers, and temporal deictic commands. Notably, the system operates under a prompt–command language mismatch, accurately interpreting Korean commands while being guided by English-based prompts. Analysis across progressive system configurations further demonstrates that structured prompting plays a critical role in reasoning performance. These results highlight the effectiveness of a prompt-driven approach for deictic interpretation and spatio-temporal grounding, providing a practical training-free framework for HRI.