Hybrid multimodal method for workers’ activity fatigue detection featuring human activity recognition

Work-related musculoskeletal disorders (WMSDs) are among the most common occupational diseases, resulting from prolonged engagement in physically demanding tasks such as manual material handling (MMH). Physical fatigue is a primary risk factor contributing to the onset of WMSDs. Hence, monitoring and estimating fatigue during task execution is crucial for effective risk mitigation. This study proposes a novel hybrid multimodal system architecture, termed hybrid multimodal fatigue estimation algorithm (HMFEA), which integrates a kinematic model, artificial intelligence (AI), and a physiological model in a hierarchical cascade framework to model and estimate physical fatigue. The novel HMFEA architecture relies on a hierarchical approach in which modules are arranged in a sequential order. A set of inertial measurement units integrated into a proprietary smart clothing, the Smart Suit provides the algorithm with continuous data streaming. Two neural network models, trained on data from 14 participants performing various MMH tasks in a controlled laboratory environment, enable task recognition. Then, a physiology-inspired model based on metabolic energy consumption allows for fatigue estimation. This cascading design offers modularity by enabling independent development of each component and improved interpretability by combining physics- and physiology-based models with data-driven learning in near-real-time operation. The HMFEA architecture is further validated on five construction workers during relevant tasks on site. AI-based activity recognition achieved 94.6% accuracy, while the HMFEA estimates fatigue with a mean absolute error of 7.1% against workers’ self-reported questionnaires. The proposed system demonstrates high accuracy and continuous fatigue estimation during work activities, resulting in a valid method to improve workplace safety while preventing injuries.