Leveraging Attention Mechanisms in Vision Transformer for Human Activity Recognition

Due to the emergence of AI, human activity recognition (HAR) domain is now one of the widely explored challenges expanding its necessity in the field of action data analysis, sports data analysis, fall detection, crime activity recognition, and many more. Yet HAR still experiences many challenges due to environmental dependency, different viewpoints, and illumination conditions making handling inter-class variance and intra-class similarity more challenging. As traditional approaches often rely on the spatial feature extractor for spatial context learning and the temporal feature extractor for training on the temporal domain, without the help of the temporal feature extractor, the spatial feature extractor often fails to generalize the model due to dependency or biases on action environmental aspect. Such a problem arises due to the fact that spatial feature extractors are trained to recognize action by learning to associate a particular background with a specific action using spatial correlation. Although debiasing techniques such as debiasing networks, classifiers, and cascading methods were implemented but could not focus on the specific relation on human or object identifier techniques. In such cases the necessity for implementing an effective spatial feature attention for image data or spatio-temporal attention mechanism for video data are often widely unexplored. The model may concentrate on the most pertinent portions of the input data thanks to attention methods. Based on the limitations of previous work we propose a vision transformer leveraging spatial attention mechanism in image data and spatio-temporal attention in video data-based architecture where we have evaluated the model using real-world datasets UCF Sports Action and BAR dataset getting max accuracy of 96.66% and 64.11% outperforming other state-of-the-art methods.