Explainable multimodal hand gesture recognition using time-series image fusion and deep residual learning

Gesture recognition remains a challenging task in human-computer interaction due to complex temporal dynamics and high intra-class variability. Traditional approaches often suffer from limited feature representation capabilities and lack of model interpretability, particularly when dealing with large gesture vocabularies in noisy environments. To address these limitations, this study presents an explainable multimodal gesture recognition framework that integrates time-series image fusion with deep residual learning to enhance both accuracy and interpretability. Three complementary image transformation techniques, recurrence plots (RP), Markov transition fields (MTF), and Gramian angular fields (GAF), are employed to convert one-dimensional temporal gesture sequences into two-dimensional representations, capturing diverse spatiotemporal characteristics. These image modalities are fused with the original time-series data to form enriched multimodal feature sets, which are processed by a customized deep residual network for end-to-end classification. Comprehensive experiments conducted on a self-constructed dataset collected under controlled indoor conditions containing 62 gesture classes (10 digits and 52 English letters) demonstrate recognition accuracies of 100%, 100%, and 99% for Arabic numerals, uppercase letters, and lowercase letters, respectively, achieving an overall accuracy of 99%, which represents a 5.5% improvement over conventional unimodal approaches. Furthermore, SHapley Additive exPlanations (SHAP) analysis provides quantitative interpretability by elucidating individual feature contributions, enhancing transparency and model trustworthiness. The proposed framework thus delivers a significant advancement in gesture recognition by simultaneously achieving high precision, multimodal robustness, and explainable decision-making.