Inertial Strengthened CLIP model for Zero-shot Multimodal Egocentric Activity Recognition

The popular CLIP model has empowered various zero-shot learning tasks by unifying them into the vision-language alignment framework. However, due to the dynamic subject-object interactions and complex motion variations, the egocentric videos become more diverse and widen the gap between the vision and language data, which limits the applicability of the CLIP model. Thanks to the widely used Inertial Measurement Unit (IMU) in wearable devices, this paper proposed an Inertial Strengthened CLIP (IS-CLIP) model to refine the visual representation of egocentric videos, which achieves highly effective zeroshot multimodal egocentric activity recognition. Our IS-CLIP is composed of two modules, i.e., the Subject-object Interaction Refinement Module (SIRM) and the Subject-motion Guided Aggregation Module (SGAM). On the one hand, SIRM embeds the motion patterns of IMU into the visual representation of each frame to enhance the action-related features. On the other hand, the SGAM maps the IMU to the frame-wise weights to aggregate the visual representations of all frames, which reduces the differences between the videos of the same action with different speeds, directions, and temporal locations. Experiments on the UESTC-MMEA-CL dataset show that the proposed IS-CLIP outperforms many state-of-the-art methods in the zero-shot multimodal egocentric activity recognition task.