Knee osteoarthritis screening using multimodal gait signals transformed via Gramian angular field and integrated by a deep learning model

To address the prevailing challenges associated with the screening of knee osteoarthritis (KOA), which include the high costs associated with imaging technologies, intricate procedural requirements, and the lack of dynamic functional information, this study introduces a multimodal gait analysis approach utilizing wearable inertial measurement units. This approach involves the conversion of time-series gait data into corresponding Gramian Angular Field (GAF) images. A dual-channel architecture was developed, integrating temporal convolutional networks (TCNs) and depth-wise separable convolutional neural networks, with multimodal feature fusion facilitated by a multi-head attention (MHA) mechanism. The experimental results demonstrated that the proposed model achieved an accuracy of 97.87%, a precision of 98.23%, a recall of 98.17%, and an F1-score of 98.19% in ten-fold cross-validation on our dataset, outperforming various established time-series models and single-modal approaches. This study substantiates that integration of GAF images within a multimodal framework significantly improves screening sensitivity and robustness, with the characteristics of high accuracy, cost-effectiveness, and radiation-free operation.