Query-Based Sign Spotting in Continuous Sign Language Videos

Sign language is a vital mode of communication for the Deaf and Hard of Hearing (DHH) community, conveying meaning through hand gestures, facial expressions, and body movements. The increasing need for automated sign language processing drives the development of methods like sign spotting. In this work, we propose a methodology for sign spotting in continuous Indian Sign Language (ISL) videos, utilizing an Inception Inflated 3D Convolutional Neural Network (Inception I3D ConvNet) model combined with Dynamic Time Warping (DTW) demonstrated on a subset of the ISLRTC Dictionary Dataset. We train the model on isolated dictionary-style ISL videos that are processed to extract feature sets, which encapsulate the signing motion of each word and serve as a reference for testing. A query word is provided, and its corresponding feature representation is matched against segments of continuous ISL videos using DTW. The best-matching subsequence is identified, and the corresponding start and end frames of the sign are retrieved. For evaluation, performance metrics such as F1-score and Mean Intersection over Union (mIoU) are used to assess the accuracy of the detected frame range. The method locates specific signs in continuous ISL videos, showing results with an F1-score of 57.34 and mIoU of 55.27. This work aims to enable more efficient sign language processing, fostering enhanced accessibility and real-time applications for the DHH community.