Time‐Frequency Transformer: A Network for Sensor‐Based Human Activity Recognition

ABSTRACT Sensor‐based Human Activity Recognition (HAR) has consistently remained a prominent research focus in domains such as human‐object interaction, health monitoring, and healthcare. In recent years, diverse deep learning models (DLM) such as CNN, LSTM, and Transformer have garnered significant interest from researchers as an effective solution for HAR. Nevertheless, a significant research gap persists: although most existing approaches primarily emphasize temporal dependency, they have not sufficiently explored two critical aspects—frequency domain features and spatial relationships among sensor axes. These elements can serve as natural data augmentation and provide complementary information that is crucial for distinguishing similar activities. Therefore, building upon previous work, we attempt to address this gap by strengthening the incorporation of frequency dependency and spatial information, proposing a more robust and accurate DLM for HAR—termed the Time‐Frequency Transformer (TF Transformer). TF Transformer performs shallow feature extraction by leveraging the time‐frequency dependency of sensor signals in the main channel. It then employs stacked Transformer encoders for deep feature modeling, and finally aggregates the time‐over‐sensor features extracted from the auxiliary channel to accomplish human activity recognition. Additionally, our network model integrates attention mechanism to assign weights to each sensor axis, enabling the accommodation of multimodal data and amplification of valuable information to enhance classification and recognition accuracy. Our method is evaluated on three publicly available datasets, namely PAMAP2, Opportunity, and Skoda, achieving Macro F1‐scores of 0.99, 0.73, and 0.96, respectively. Extensive experimental results demonstrate that, compared to state‐of‐the‐art approaches, our model exhibits higher accuracy and superior robustness.