Img2Acoustic: A Cross-Modal Gesture Recognition Method Based on Few-Shot Learning

Acoustic-based human gesture recognition (HGR) offers diverse applications due to the ubiquity of sensors and touch-free interaction. However, existing machine learning approaches require substantial training data, making the process time-consuming, costly, and labor-intensive. Recent studies have explored cross-modal methods to reduce the need for large training datasets in behavior recognition, but they typically rely on open-source datasets that closely align with the target domain, limiting flexibility and complicating data collection. In this paper, we propose <inline-formula><tex-math notation="LaTeX">${\sf Img2Acoustic}$</tex-math></inline-formula>, a novel cross-modal acoustic-based HGR approach that leverages models trained on open-source image datasets (i.e., EMNIST, Omniglot) to effectively recognize custom gestures detected via acoustic signals. Our model incorporates a task-aware attention layer (TAAL) and a task-aware local matching layer (TALML), enabling seamless transfer of knowledge from image datasets to acoustic gesture recognition. We implement <inline-formula><tex-math notation="LaTeX">${\sf Img2Acoustic}$</tex-math></inline-formula> on commercial devices and conduct comprehensive evaluations, demonstrating that our method not only delivers superior accuracy and robustness compared to existing approaches but also eliminates the need for extensive training data collection.