GameSense: hierarchical spatio-temporal transformer for basketball player tracking and tactical performance analysis

The analysis of basketball gameplay through multi-object tracking and action recognition is pivotal for enhancing player performance, tactical planning, and audience engagement. However, existing methods often suffer from limitations, including reliance on bounding box annotations, insufficient handling of occlusions, and a lack of holistic scene understanding. These limitations hinder the scalability and robustness of traditional approaches, particularly in dynamic and complex sports environments. To address these challenges, we propose a comprehensive framework combining the Basketball Player Tracking Network (BPTN) and the Basketball Player Analytics Network (BPAN). The BPTN employs hierarchical temporal memory and transformer-based architecture to ensure precise player tracking, robust identity association, and seamless handling of occlusions. The BPAN leverages multi-scale vision transformers and hierarchical temporal processing for accurate classification of basketball-specific maneuvers. The framework introduces novel modules, including a candidate region module for efficient player detection, a Long-Term Context Buffer for maintaining player identities, and an action proposal module that captures spatio-temporal encodings for effective maneuver recognition. Our approach eliminates the dependency on bounding box annotations during inference and incorporates contextual scene-level features, significantly improving scalability and performance. On the SportsMOT dataset, BPTN achieves tracking performance with HOTA of 81.6, AssA of 75.0, and a reduction in identity switches. Similarly, BPAN outperforms existing action recognition models on the Basketball-51 dataset, achieving an accuracy of 92.76%, precision of 92.06%, recall of 91.75%, and an F1-score of 91.74%. These results underscore the robustness and applicability of the proposed framework in capturing complex player dynamics and gameplay scenarios, paving the way for advanced sports analytics solutions.