Action and Agent Recognition from Situational Images Using Transformer-based Approach

In an increasingly digitized world, the automated recognition of activities and agents from images is pivotal for many applications, such as surveillance systems, social media monitoring, sports analysis, smart home systems, elderly care, action understanding by robots, etc. This work delves into activity recognition and agent detection from situational images, addressing the emerging field of single-image activity recognition. The work presents a novel dataset comprising 5,465 annotated images for action recognition and 6,987 bounding box annotations for object detection. The efficacy of the proposed approach is demonstrated through a rigorous evaluation process involving different techniques. The proposed methodology uses the ViT-4 model for action recognition and the detection transformer (DETR) for agent detection. The evaluation results show that the ViT-4 model gained a weighted F1 score of 0.883 (for action recognition), and DETR’s mAP model obtained 0.796 (for agent detection), exceeding existing techniques.