ST-AMMFN: a Spatio-Temporal Attention-based Multi-Modal Fusion Network for gait data simulation and rehabilitation effect prediction in ankle injury patients

This paper presents a novel deep learning framework for gait data simulation and rehabilitation effect prediction in patients with ankle injuries. We propose the Spatio-Temporal Attention-based Multi-Modal Fusion Network (ST-AMMFN), which effectively integrates heterogeneous data from multiple sources including demographic information, clinical assessments, biomechanical measurements, and wearable sensor time-series data. The ST-AMMFN architecture features specialized encoders for different data modalities, a hierarchical attention mechanism that captures both temporal dynamics and modality importance, and a multi-task prediction structure that simultaneously forecasts rehabilitation progress, ultimate recovery level, and expected rehabilitation duration. We developed ST-AMMFN using real clinical data from 300 ankle injury patients for both rehabilitation outcome prediction and synthetic gait data generation to augment training datasets. Experimental results demonstrate that our proposed model significantly outperforms traditional machine learning approaches and state-of-the-art deep learning methods, achieving an RMSE of 0.219 and R2 of 0.871 in rehabilitation effect prediction, representing 13.8% lower RMSE and 5.2% higher R2 compared to the best baseline methods including traditional machine learning approaches (Random Forest, SVR, GBM), deep learning methods (MLP, CNN, LSTM, TCN), and multi-modal fusion approaches (Early Fusion, Late Fusion, NeuMF, MMTM, MMFNet). The interpretability provided by the attention mechanism further allows clinicians to identify critical time points and influential data modalities in the rehabilitation process, potentially enabling more personalized intervention strategies. Our approach presents a promising direction for integrating artificial intelligence into rehabilitation medicine, particularly for objective assessment and prediction of functional recovery trajectories in musculoskeletal injuries.