A dual-branch keypoint Transformer for continuous Korean Sign Language recognition

Continuous Korean Sign Language recognition requires practical models that capture manual and non-manual articulation. We propose KDS-Former, a compact dual-branch keypoint Transformer that encodes hand and body–face motion, fuses the temporal representations, and predicts gloss sequences using connectionist temporal classification. On KSL-Guide, KDS-Former-S achieved word error rates (WERs) of \(6.33 \pm 1.68\) % and \(7.61 \pm 2.09\) % under seen-signer/unseen-sentence and unseen-signer/unseen-sentence protocols, respectively, using 2.59 million parameters and 1.91 model-compute GFLOPs. KDS-Former-M achieved \(5.68 \pm 1.98\) % and \(6.27 \pm 2.24\) % using 14.75 million parameters and 9.93 model-compute GFLOPs. The larger systems of Min et al. and TEMPO achieved lower mean WERs across three runs, whereas KDS-Former required fewer parameters, lower model-compute GFLOPs, and lower peak GPU memory usage. Under both primary protocols, the full KDS-Former-S achieved lower mean WER than all five controlled ablation variants, although the unseen-signer/unseen-sentence single-branch result showed substantial between-run variability. Under controlled input corruption, both KDS-Former variants showed smaller mean WER degradation than ST-GCN. These results position KDS-Former as an efficiency-oriented, model-level alternative rather than an accuracy-leading method. Signer-generalization results are limited by evaluation on only two held-out signers, and the reported latency was measured on an RTX 4090 without upstream keypoint extraction, preprocessing, or CTC decoding; broader signer-disjoint and hardware-specific validation is therefore required.

Authors

Institutions

Publication Details

Journal
Scientific Reports
Published
2026-11-16
DOI
https://doi.org/10.1038/s41598-026-75669-9
Primary Topic
Hand Gesture Recognition Systems
Type
article
Field-Weighted Citation Impact
0.00

Funders

Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

A dual-branch keypoint Transformer for continuous Korean Sign Language recognition

Hyunsoo Kang, Dohyeon Kim
3 links
Scientific Reports
Hand Gesture Recognition Systems
article

A dual-branch keypoint Transformer for continuous Korean Sign Language recognition

Hyunsoo Kang, Dohyeon Kim
article en

Abstract

Continuous Korean Sign Language recognition requires practical models that capture manual and non-manual articulation. We propose KDS-Former, a compact dual-branch keypoint Transformer that encodes hand and body–face motion, fuses the temporal representations, and predicts gloss sequences using connectionist temporal classification. On KSL-Guide, KDS-Former-S achieved word error rates (WERs) of \(6.33 \pm 1.68\) % and \(7.61 \pm 2.09\) % under seen-signer/unseen-sentence and unseen-signer/unseen-sentence protocols, respectively, using 2.59 million parameters and 1.91 model-compute GFLOPs. KDS-Former-M achieved \(5.68 \pm 1.98\) % and \(6.27 \pm 2.24\) % using 14.75 million parameters and 9.93 model-compute GFLOPs. The larger systems of Min et al. and TEMPO achieved lower mean WERs across three runs, whereas KDS-Former required fewer parameters, lower model-compute GFLOPs, and lower peak GPU memory usage. Under both primary protocols, the full KDS-Former-S achieved lower mean WER than all five controlled ablation variants, although the unseen-signer/unseen-sentence single-branch result showed substantial between-run variability. Under controlled input corruption, both KDS-Former variants showed smaller mean WER degradation than ST-GCN. These results position KDS-Former as an efficiency-oriented, model-level alternative rather than an accuracy-leading method. Signer-generalization results are limited by evaluation on only two held-out signers, and the reported latency was measured on an RTX 4090 without upstream keypoint extraction, preprocessing, or CTC decoding; broader signer-disjoint and hardware-specific validation is therefore required.

Scientific Reports
Chungbuk National University (KR)
Ministry of Education, National Research Foundation of Korea, Ministry of Science and ICT, South Korea, Institute for Information and Communications Technology Promotion, Division of Human Resource Development
Hand Gesture Recognition Systems
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon