A Deep Feature–Based Dynamic Hand Gesture Recognition Framework for Word and Sentence Formation Using Mediapipe

Individuals who are deaf or experience difficulties with hearing and speech predominantly rely on sign language as their medium to communicate, which is not universally comprehended leading to obstacles in achieving effective communication. Advances in deep learning technologies in recent years have enabled the development of systems intended to autonomously interpret gestures in sign language and translate them into spoken language. This paper introduces a system built on deep learning methodologies for recognizing sign language. It uses long short-term memory (LSTM) architecture to distinguish and classify hand gestures which are static and dynamic. The system is divided into three primary components, including dataset collection, neural network assessment, and sign detection component that encompasses hand gesture extraction and sign language classification. The project was divided into two major phases: Static Sign Recognition and Dynamic Sign Recognition. During the static phase, a fully original dataset comprising 120,000 images was created for 24 alphabets (A–Z excluding J and Z). Each image was then treated through two custom-designed filters: silhouette extraction and finger detection, carried out using MediaPipe for precise annotation and cropping on its own. The training of the static model happened with custom-designed CNN applied on the data to achieve accuracies of 94% (finger detection), 89% (silhouette), and 91% (combined). In the dynamic phase, signs with movement were captured using MediaPipe landmarks and registered as NumPy arrays (CSV and .NPY) files. Hand and facial keypoints were stored at every frame for temporal learning. This phase included one-hand, two-hand, and hand-with-face categories. The whole system achieved 15–20 FPS real-time inference on CPU alone. While the static signs and single-handed dynamic signs were performing very well, the accuracy of complex two-handed and face-assisted gestures suffered due to scarcity and noise in the data. The results provide a very good baseline to improve upon for the future using deep temporal networks like LSTM or 3D CNNs.

Authors

Publication Details

Journal
The International Journal of Maritime Engineering
Published
2026-11-13
DOI
https://doi.org/10.5750/ijme.v168ia6(s).2022
Primary Topic
Hand Gesture Recognition Systems
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

A Deep Feature–Based Dynamic Hand Gesture Recognition Framework for Word and Sentence Formation Using Mediapipe

Siddangouda Hosamani, Jithendra P. R. Nayak
4 links
The International Journal of Maritime Engineering
Hand Gesture Recognition Systems
article

A Deep Feature–Based Dynamic Hand Gesture Recognition Framework for Word and Sentence Formation Using Mediapipe

Siddangouda Hosamani, Jithendra P. R. Nayak
article en

Abstract

Individuals who are deaf or experience difficulties with hearing and speech predominantly rely on sign language as their medium to communicate, which is not universally comprehended leading to obstacles in achieving effective communication. Advances in deep learning technologies in recent years have enabled the development of systems intended to autonomously interpret gestures in sign language and translate them into spoken language. This paper introduces a system built on deep learning methodologies for recognizing sign language. It uses long short-term memory (LSTM) architecture to distinguish and classify hand gestures which are static and dynamic. The system is divided into three primary components, including dataset collection, neural network assessment, and sign detection component that encompasses hand gesture extraction and sign language classification. The project was divided into two major phases: Static Sign Recognition and Dynamic Sign Recognition. During the static phase, a fully original dataset comprising 120,000 images was created for 24 alphabets (A–Z excluding J and Z). Each image was then treated through two custom-designed filters: silhouette extraction and finger detection, carried out using MediaPipe for precise annotation and cropping on its own. The training of the static model happened with custom-designed CNN applied on the data to achieve accuracies of 94% (finger detection), 89% (silhouette), and 91% (combined). In the dynamic phase, signs with movement were captured using MediaPipe landmarks and registered as NumPy arrays (CSV and .NPY) files. Hand and facial keypoints were stored at every frame for temporal learning. This phase included one-hand, two-hand, and hand-with-face categories. The whole system achieved 15–20 FPS real-time inference on CPU alone. While the static signs and single-handed dynamic signs were performing very well, the accuracy of complex two-handed and face-assisted gestures suffered due to scarcity and noise in the data. The results provide a very good baseline to improve upon for the future using deep temporal networks like LSTM or 3D CNNs.

The International Journal of Maritime EngineeringVol. 168(A6(S))
Hand Gesture Recognition Systems
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon