CNN and ViT as Dual Teachers for Knowledge Distillation in Human Action Recognition

Recognizing actions in a single image remains challenging, even if Convolutional Neural Networks (CNNs) have achieved great progress in picture categorization. The primary challenge in recognizing human action from photographs is the lack of any visual indications regarding temporal information in static photos. The current best practices involve training a Deep Convolutional Neural network (DCNN) directly on images for an action recognition task. Nevertheless, these approaches include numerous parameters and incur substantial computational expenses. Moreover, the majority of existing techniques have utilized supplementary information, such as human body movements, relevant objects, and the visual characteristics of body parts, obtained from photos. These approaches utilize either object detection or pose estimation as an auxiliary task in both the training and test stages. Creating these kinds of annotations is very time-consuming and therefore expensive. This paper leverages a Convolutional Neural Network (CNN) as the student model, while a larger CNN combined with a Vision Transformer (ViT) act as the teacher model. The CNN teacher gathers local features of the picture, while the ViT teacher emphasizes global features via the attention mechanism. Both local and global features can be advantageous in human action recognition; therefore, we attempt to integrate ViT and CNN teachers to develop the optimal teacher for the student model. This is achieved through a loss-aware module, which adaptively allocates sample-wise reliability to each teacher's prediction based on ground-truth labels, assigning greater weights to predictions that closely match one-hot labels. Our method achieves a mean Average Precision (mAP) of 95.17% on the Stanford40 dataset, which is the highest performance.