A dual-branch keypoint Transformer for continuous Korean Sign Language recognition
Continuous Korean Sign Language recognition requires practical models that capture manual and non-manual articulation. We propose KDS-Former, a compact dual-branch keypoint Transformer that encodes hand and body–face motion, fuses the temporal representations, and predicts gloss sequences using connectionist temporal classification. On KSL-Guide, KDS-Former-S achieved word error rates (WERs) of \(6.33 \pm 1.68\) % and \(7.61 \pm 2.09\) % under seen-signer/unseen-sentence and unseen-signer/unseen-sentence protocols, respectively, using 2.59 million parameters and 1.91 model-compute GFLOPs. KDS-Former-M achieved \(5.68 \pm 1.98\) % and \(6.27 \pm 2.24\) % using 14.75 million parameters and 9.93 model-compute GFLOPs. The larger systems of Min et al. and TEMPO achieved lower mean WERs across three runs, whereas KDS-Former required fewer parameters, lower model-compute GFLOPs, and lower peak GPU memory usage. Under both primary protocols, the full KDS-Former-S achieved lower mean WER than all five controlled ablation variants, although the unseen-signer/unseen-sentence single-branch result showed substantial between-run variability. Under controlled input corruption, both KDS-Former variants showed smaller mean WER degradation than ST-GCN. These results position KDS-Former as an efficiency-oriented, model-level alternative rather than an accuracy-leading method. Signer-generalization results are limited by evaluation on only two held-out signers, and the reported latency was measured on an RTX 4090 without upstream keypoint extraction, preprocessing, or CTC decoding; broader signer-disjoint and hardware-specific validation is therefore required.
Authors
- Hyunsoo Kang
- Dohyeon Kim
Institutions
- Chungbuk National University (KR)
Publication Details
- Journal
- Scientific Reports
- Published
- 2026-11-16
- DOI
- https://doi.org/10.1038/s41598-026-75669-9
- Primary Topic
- Hand Gesture Recognition Systems
- Type
- article
- Field-Weighted Citation Impact
- 0.00
Funders
- Ministry of Education
- National Research Foundation of Korea
- Ministry of Science and ICT, South Korea
- Institute for Information and Communications Technology Promotion
- Division of Human Resource Development