Comparative Analysis of Training and Decoding Techniques in Transformer-Based Models for Indian Sign Language Recognition and Translation

Sign language users often face communication barriers in public spaces, limiting access to essential services and social interaction. This research presents an Indian Sign Language (ISL) Recognition and Translation System leveraging Transformer-based architectures—VideoMAE for feature extraction and a Custom Transformer for sequence decoding. The model processes video inputs, extracts spatiotemporal embeddings, and translates gestures into meaningful English text.To improve recognition accuracy, different training strategies were explored, including Teacher Forcing, Inference-based Training, and Scheduled Sampling. The impact of these methods was evaluated using BLEU, ROUGE-L, and METEOR scores, demonstrating how each approach affects translation quality. Additionally, various decoding techniques—Greedy Search, Multinomial Sampling, and Top-K Sampling—were compared to determine their influence on output fluency and accuracy.A key innovation of this system lies in its optimized tokenizer, which ensures better alignment with ISL’s unique grammar and sequential structure. Experimental results show that the proposed architecture achieves 99.78% accuracy on the evaluated dataset, highlighting its effectiveness in handling ISL’s linguistic complexities. By integrating different training and decoding strategies, this study provides valuable insights into enhancing ISL recognition systems for real-world applications.