Dual-Branch Framework for Panoramic Activity Recognition: Integrating Appearance and Trajectory Information for Enhanced Social Relationship Modeling

Panoramic activity recognition is a novel task aimed at simultaneously identifying behaviors at multiple granularities within the same framework, including individual actions, social group activities, and global activities, without restricting the number of individuals or subgroups. However, existing methods primarily rely on appearance features and spatial positions within a single frame, neglecting the dynamic interactions captured by trajectory information, which limits their ability to accurately model complex social relationships. To address these challenges, we propose a dual-branch framework that integrates appearance and trajectory information to comprehensively model social relationships. In the appearance branch, we capture the spatiotemporal context of appearance features through spatiotemporal self-attention mechanism. In the trajectory branch, we use the OSPA <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">(2)</sup>as a trajectory similarity metric and construct a graph transformer with LSTM-extracted trajectory features as nodes to effectively model the dynamic inter-trajectory interactions between individuals. We design a cross-granularity aggregation module based on the transformer encoder to capture unified motion patterns through a shared aggregation module, which then generates features for social groups and global activities. Compared to previous methods that only rely on appearance features and spatial positions within a single frame, the proposed method is able to more comprehensively capture social relationships and behavioral patterns in complex scenarios.