Fatigue State Recognition System for Miners Based on a Multimodal Feature Extraction and Fusion Framework

The fatigue factor is widely recognized as a primary contributor to accidents in the mining industry. Proactively recognizing fatigue states in miners before starting work can effectively establish a safety boundary for both miners safety and coal mine production. Therefore, this study designs a fatigue state recognition system for miners based on a multimodal extraction and fusion framework. First, the system is equipped with various sensors, a core processor and a display to collect and process physiological data such as electrocardiogram (ECG), electrodermal activity (EDA), blood pressure (BP), blood oxygen saturation (SpO<inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">${}_{2}$</tex-math></inline-formula>), skin temperature (SKT), as well as facial data, and to present fatigue state, respectively. Second, based on the multimodal feature extraction and fusion framework, after the necessary preprocessing steps, the system extracts physiological features by time and frequency domain analysis, extracts facial features by ResNeXt-50 and gated recurrent unit (GRU), and fuses multifeatures by Transformer+. Finally, in the comprehensive laboratory for coal-related programs of Xi’an University of Science and Technology, we test the system and build a multimodal dataset, and the results demonstrate an average accuracy of 93.15%.