Towards Robust Identity Incorporation in Sports Video Captioning Systems

Sports video analysis is a rapidly growing field. Yet, identity-aware, temporally grounded captioning, linking global player identities across fast, multi-agent interactions, remains underexplored. We address this gap with a novel method for integrating player identity into multimodal sports video models, improving captioning, action understanding, and player recognition. Our approach (1) employs triangular attention masking within modality encoders, capturing temporal inductive biases to better model action sequences and their causal flow, (2) proposes player token injection for global identity grounding, enabling the model to connect visual observations to named individuals, and (3) simulates ball possession sequences, mimicking real-world tracking data to strengthen the link between actions and involved players. Both combined and individually, each of these components allows us to significantly improve caption quality and player classification accuracy, as well as enhance the temporal comprehension in our multimodal data. Using extensive experimentation, we show that our method achieves substantial improvements over prior work (e.g., up-to 225% for some video captioning tasks and over 14× for some player recognition tasks), generalize to other domains, and provide insights into the best design tradeoffs. The results highlight a promising avenue for automated understanding and interpretation of dynamic sports content.