Impact of introducing sparse inertial Measurement Units in Computer Vision-Based Motion Capture Systems for Ergonomic Postural Assessment

In ergonomics, considering a worker's movement is important for assessing the risks of musculoskeletal disorders, among many other factors. Several commercial motion capture systems are available, mostly based on monocular or multi RGB (THEIA system www.theiamarkerless.com) or RGB-D videos (Microsoft Kinect system). Hybrid systems combining computer vision and Inertial Measurement Units (IMUs) have been introduced, such as the KIMEA (1 RGB-D+4 IMUs) and the KIMEA Cloud (1 RGB+4 IMUs) solutions (www.Moovency.com). Although previous works analyzed the accuracy of some of these systems, the relevance of coupling computer vision and IMU has not been studied. Hence, we tested the performance of these systems in measuring bimanual handling tasks, which lead to partial occlusions of the body in the images. As in previous works, the Xsens (www.movella.com) system was used as a reference (Kim et al., 2021), because it is not affected by such occlusions. Three Orbbec depth cameras were installed around the participant with different viewpoints. Six RGB cameras were also placed around the subject. Additional inertial (IMU) sensors were placed over both arms and the forearms, as recommended by the KIMEA and KIMEA Cloud systems. Testing several camera viewpoints makes sense in real industrial conditions where the camera placement is generally strongly constrained. 12 participants, 3 women and 9 men (age: 32.6 +/- 10 years, height: 1.73 +/- 0.079 m, mass: 76 +/- 16 kg) performed the following task: removing an empty cardboard box (size: 39x29.5x19cm, weight: 250g) from a three-tier shelf and transferring it to another one, repeated five times. A reference pose (corresponding to 0 value for each degree of freedom) was preliminary performed for each subject. It enabled us to estimate the same angular offsets for each pose, compared to this reference pose. As a results, angles measured by the various systems have similar definitions and can be compared. This study was approved by the Operational Committee for the Evaluation of Legal and Ethical Risks (COERLE) No. 2021-32.Results for the computer vision methods only are similar to those reported in Kim et al. (2021) and Lakhar et al. (2022). The THEIA system exhibits an average of 11.1° error for all the joints, with larger Root Mean Square errors on the wrists and the shoulder (>14° error). KIMEA Cloud with IMU obtained similar global RMS error (10.3 ° to 10.9° depending on the viewpoint), but with obviously better results for the wrists (3.9 ° to 4.3°). The impact of coupling RGB-D images and IMU data is even bigger: the RMS error of the Kinect decreased from 17.2° to 8.9° when adding the IMUs information (KIMEA system). This difference is even bigger for the wrists: 28.3° to 38.5° for the Kinect, and 3.8 ° to 4° for KIMEA. When studying the Mean Absolute Error, or the Spearman’s correlation coefficient, these statements are consistent. These results confirm the advantage of introducing a few IMU sensors, especially for the wrists which are badly tracked in the images: often occluded and consisting in a little number of pixels in the image. We also show that it leads to more robust measurement for various points of views, including from the back. The information provided by the IMU, not impacted by the occlusions, seems to help the computer vision system to reconstruct the whole-body posture. However, adding IMUs also introduces some experimental constraints compared to simply placing one camera. THEIA obtained the best results for the whole-body reconstruction, but still had difficulties to correctly reconstruct the wrists angles. On the one hand, THEIA does not involve to add sensors. On the other hand, it involves the calibration of a multi-camera system, which might be difficult to place in real industrial environment. References Kim, W., Sung, J., Saakes, D., Huang, C., Xiong, S., 2021. Ergonomic postural assessment using a new open-source human pose estimationtechnology (openpose). International Journal of Industrial Ergonomics 84, 103164. doi:https://doi.org/10.1016/j.ergon.2021.103164Lahkar, B.K., Muller, A., Dumas, R., Reveret, L., Robert, T., 2022. Accuracy of a markerless motion capture system in estimating upper extremitykinematics during boxing. Frontiers in Sports and Active Living 4, 939980. doi:https://doi.org/10.3389/fspor.2022.939980.