Temporal Contrastive Learning for Sensor-Based Human Activity Recognition: A Self-Supervised Approach

Deep learning techniques can make use of a large amount of time-series data from wearable devices and greatly benefit the development of sensor-based human activity recognition (HAR). However, representation learning in a supervised manner requires massive labeled sensory data that are time-consuming to obtain and hindered by privacy concerns. To address these issues, we utilize the plentiful unlabeled sensory data and propose a novel self-supervised learning framework, namely temporal contrastive learning in HAR (TCLHAR), which learns the meaningful feature representations for time-series data without labels. Our TCLHAR framework utilizes the temporal co-occurrence relationship among time windows as the supervisory signals to construct positive pairs in the encoder pretraining stage. The encoder is designed for cross-modality fusion, which leverages the local interactions of each sensor modality and the global fusion of features from different sensors. The proposed framework is extensively evaluated on public HAR datasets in supervised, self-supervised, and semi-supervised settings. Our method outperforms several self-supervised learning benchmark models, achieving comparable results with fully labeled data training. When labeled data are scarce, our method can boost the <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">${F}1$ </tex-math></inline-formula> score by up to 65% over traditional supervised training, which demonstrates the effectiveness of our feature representations.