Enhancing clinical gait abnormality analysis with multimodal deep learning

Abstract Advances in computer vision have produced promising results in gait analysis, which plays an important role in clinical diagnosis. These technologies enable early and objective detection of diseases by identifying changes in individual walking patterns. This study investigates multimodal clinical gait abnormality detection on the Gait Abnormality Video Dataset (GAVD) by combining appearance features and skeleton-based structural motion features. SlowFast was used to extract spatiotemporal appearance representations from RGB videos, while ST-GCN was used to model skeleton joint sequences. The extracted features were combined using late fusion and classified with an MLP classifier. The proposed framework was evaluated on the GAVD dataset using 5-fold stratified cross-validation. The results show that skeleton-based representations carry a substantial part of the discriminative signal, while RGB features provide complementary information when extracted by a sufficiently strong video backbone. The proposed SlowFast + ST-GCN model achieved a recall of 0.947, an F1-score of 0.910, and an AUC of 0.783. Under the abnormal-only test protocol, the same model achieved 95.0% accuracy. Ablation experiments showed that concatenation was the most effective fusion strategy, while frozen feature extraction with MLP classification provided the best generalization among the evaluated training strategies. Comparisons with GAVD-adapted multimodal baselines further showed that different modality combinations offer different strengths. Overall, this study provides a systematic multimodal evaluation on GAVD and shows that RGB + skeleton fusion is useful for abnormal gait detection.