Vision-language models meet hand gesture recognition: towards multimodal and intelligent human-computer interaction

Hand Gesture Recognition (HGR) has become an important means of enhancing natural interaction with computers. However, traditional vision-only systems are limited by closed-set classification and lack firm semantic grounding. The recently introduced large vision-language models (VLMs), such as CLIP, BLIP, Flamingo, and GPT-4V, provide open-vocabulary recognition, cross-modal retrieval, and gesture captioning by learning common visual-textual representations from large image-text corpora. This survey, therefore, synthesizes approaches that fall within the general scope of attempts to inject vision-language models (VLMs) into the human-robot repertoire and neatly fits them into a three-dimensional taxonomy consisting of: learning paradigms which range from zero-shot, few-shot, prompt-based, retrieval-augmented to parameter-efficient fine-tuning; task objectives spanning classification, captioning, retrieval, and multimodal reasoning; input modalities including static images, video, RGB-D, and textual prompts. We analyze how VLMs enable the convergence of linguistic and perceptual representations toward enhanced gesture understanding, adaptability, and explainability. Building on this taxonomy, we describe the limitations of current datasets and benchmarking practices for gesture-language alignment, which manifest most significantly in cases of small datasets across linguistic diversity and varying cultural practices. We next map the open problems and research gaps toward building multimodal HGR systems that enable semantically grounded, culturally informed, and context-sensitive intelligent interactions.