FFCI: A Camera and IMU Sensors-Based Multi-Modal Neural Network for Activity Recognition in Smart Factory

Worker activity recognition is an important aspect of the construction of smart factory. The development of deep neural networks and the widespread distribution of sensors in the smart factory have brought opportunities for the recognition of workers’ activities. The existing methods based on camera and IMU sensors respectively have problems of visual occlusion and difficulty in recognition of similar activities, which result in the reduction of recognition accuracy. Therefore, we propose a feature-level fusion based on camera and IMU sensor (FFCI) for activity recognition in smart factory. The FFCI employs an optimized multi-modal fusion strategy to integrate the visual information and the action information of workers’ activities. In order to verify the effectiveness of FFCI, we define a simple yet fine-grained assembly task and collect six common operations. Furthermore, we add visual occlusion and similar activity data to get closer to the smart factory. Finally, we evaluate the FFCI on the collected dataset and achieve a recognition accuracy of 98.17%, demonstrating its effectiveness in accurately classifying workers’ activities.