Hybrid deep learning approach for hand gesture recognition in classical dance

Abstract Existing Artificial Intelligence-based automatic mudra recognition in Indian classical dance is challenging due to intricate hand articulations, self-occlusions, and the limited ability of existing methods to jointly capture geometric and semantic information. Most of the existing methods use either deep features or handcrafted features for classification. There are still a number of issues with hand segmentation and the selection of precise features for identification. This study aims to develop an improved framework for accurate hand gesture recognition using a hybrid feature-based methodology in order to classify 28 Asamyukta mudras in Bharatanatyam. The proposed method follows feature extraction and classification of images with preprocessing, which addresses the elimination of noise from shadows and changes in illumination by applying median filtering followed by histogram equalization. U-Net is employed for accurate hand segmentation, enabling precise separation of hand contours from complex backgrounds. Hu Moment features are infused with deep features that were extracted using DNN, VGG16, and VGG19 after being enhanced using Extra Trees-based feature selection. A DNN is then used to classify the resulting hybrid feature set. Compared to individual feature-based models, the combined semantic–geometric feature representation performs better. The VGG19-based model (model 3) had the highest classification accuracy of 95%, according to the experimental data, followed by the VGG16 model (model 1) with 93% and the DNN model (model2) with 91%. The results show how effectively the hybrid features perform in a mudra identification system. The suggested approach provides significant potential for digitally learning, documenting, and preserving Indian cultural dances.