A Machine Learning-Based Fatigue Extraction Method Using Human Manipulation Video Data in Smart Manufacturing

As an important part in smart manufacturing under Industry 4.0 era, human-robot collaboration (HRC) features the interaction between human operators and machines, which makes the research of human fatigue come into sight. However, most existing studies on human fatigue or efficiency detection are realized using detectors and models from bioelectronics, whose intrusive detection and decoding of electromyographic signal limits the generality and applicability of such methods. Therefore, this study proposes a human fatigue extraction method based on video data. A new dataset on human manipulation is established by collecting video data of assembly operations to simulate the working status of human operators under smart manufacturing environment. With human skeletal data extracted from the video using a machine learning-based pose extraction tool, MediaPipe, a spatiotemporal analysis for critical skeleton points is implemented for working status categorization and learning using a stochastic gradient descent (SGD) classifier. In this way, the duration taken to complete an assembly task can be extracted as the operation time using the trained SGD classifier, and thus the time-varying operation time series data are obtained to show the trend of human fatigue level. An accuracy of 98.3% is obtained for working status identification for the dataset. Several quantitative indicators like pearson correlation, R-squared value, root mean squared error (RMSE), and Fréchet Distance, are used to evaluate the accuracy of both the extracted operation time and its time-varying curve as compared to the ground truth, with satisfactory results showing the effectiveness of the proposed method.