Isolated Sign Language Recognition through Engineered Spatio-Temporal Landmark Features and Heterogeneous Deep Ensemble Learning

The development of systems that process and interpret human language has facilitated the rapid advancement of communication technologies with computational intelligence. However, accessibility remains limited for communities whose languages are communicated through visual and gestural modalities. Sign languages constitute sophisticated linguistic frameworks employed by millions of people globally, and the existence of sign language systems encourages the advancement of intelligent devices that can identify and interpret signed communication. This study examines the integration of heterogeneous deep learning architectures for the recognition of sign language through spatial-temporal landmark representations. We hypothesize that employing complementary spatial-temporal representations via a combination of diverse deep learning architectures results in superior and more resilient isolated sign language recognition compared to using any single model independently. To test this hypothesis, we conduct experiments on the Google Isolated Sign Language Recognition dataset, which consists of MediaPipe Holistic landmark sequences extracted from sign videos, each annotated with its corresponding American Sign Language (ASL) label. We evaluate neural architectures including convolutional recurrent networks, Transformer-based models, bidirectional recurrent networks, and hybrid CNN-Transformer models for isolated sign classification using hand and upper-body pose motion information. Our study examines the effect of feature representation choices and finds that removing unreliable depth components while preserving two-dimensional spatial coordinates improves training stability and recognition performance. An ensemble model is developed by combining complementary predictions from multiple architectures, achieving higher accuracy than individual approaches. The proposed framework achieves 72.14% classification accuracy and 94.29% top-5 accuracy on a 20-class subset of the dataset. These results demonstrate the effectiveness of landmark-based deep learning approaches for accessible sign language recognition and provide a foundation for future systems aimed at supporting more inclusive human-computer communication.

Authors

Institutions

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-18
DOI
https://doi.org/10.5281/zenodo.24326418
Primary Topic
Hand Gesture Recognition Systems
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Isolated Sign Language Recognition through Engineered Spatio-Temporal Landmark Features and Heterogeneous Deep Ensemble Learning

Ritwik Raj Saxena, Iwan Sandjaja
2 links
Zenodo (CERN European Organization for Nuclear Research)
Hand Gesture Recognition Systems
preprint

Isolated Sign Language Recognition through Engineered Spatio-Temporal Landmark Features and Heterogeneous Deep Ensemble Learning

Ritwik Raj Saxena, Iwan Sandjaja
preprint en

Abstract

The development of systems that process and interpret human language has facilitated the rapid advancement of communication technologies with computational intelligence. However, accessibility remains limited for communities whose languages are communicated through visual and gestural modalities. Sign languages constitute sophisticated linguistic frameworks employed by millions of people globally, and the existence of sign language systems encourages the advancement of intelligent devices that can identify and interpret signed communication. This study examines the integration of heterogeneous deep learning architectures for the recognition of sign language through spatial-temporal landmark representations. We hypothesize that employing complementary spatial-temporal representations via a combination of diverse deep learning architectures results in superior and more resilient isolated sign language recognition compared to using any single model independently. To test this hypothesis, we conduct experiments on the Google Isolated Sign Language Recognition dataset, which consists of MediaPipe Holistic landmark sequences extracted from sign videos, each annotated with its corresponding American Sign Language (ASL) label. We evaluate neural architectures including convolutional recurrent networks, Transformer-based models, bidirectional recurrent networks, and hybrid CNN-Transformer models for isolated sign classification using hand and upper-body pose motion information. Our study examines the effect of feature representation choices and finds that removing unreliable depth components while preserving two-dimensional spatial coordinates improves training stability and recognition performance. An ensemble model is developed by combining complementary predictions from multiple architectures, achieving higher accuracy than individual approaches. The proposed framework achieves 72.14% classification accuracy and 94.29% top-5 accuracy on a 20-class subset of the dataset. These results demonstrate the effectiveness of landmark-based deep learning approaches for accessible sign language recognition and provide a foundation for future systems aimed at supporting more inclusive human-computer communication.

Zenodo (CERN European Organization for Nuclear Research)
Petra Christian University (ID), Missouri University of Science and Technology (US)
Reduced inequalities
Hand Gesture Recognition Systems
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon