Hoop-MSSL: Multitask self-supervised representation learning on basketball spatiotemporal data

Observing and identifying on-court behaviours by basketball players, who engage in intricate spatiotemporal interactions with their teammates and the opponent players, have long been considered challenging tasks for machines. Various recent works have drawn inspiration from the field of autonomous driving to develop self-supervised learning frameworks for spatiotemporal data. However, these frameworks focus mainly on single tasks such as trajectory reconstruction or prediction and do not consider domain knowledge in basketball. In this work, we propose Hoop-MSSL, which is a multitask self-supervised representation learning framework for handling complex interactions and dependencies among spatiotemporal data. Specifically, Hoop-MSSL integrates masking augmentation and three pretraining tasks for (i) motion reconstruction, (ii) player role identification and (iii) contrastive learning to capture spatiotemporal features and role relationships across multiple dimensions. To evaluate the efficacy of Hoop-MSSL, we conducted extensive line-probing experiments on three downstream tasks. Our results demonstrate that the synergistic interaction among all the Hoop-MSSL components helps the model learn more general spatiotemporal representations, thereby enabling it to achieve better performance on all downstream tasks than when only subsets of the components are used. In particular, the contrastive learning task, in which domain knowledge is leveraged to compare different masked views of the same item and capture high-level offensive intent, proves especially effective in increasing representation quality. Finally, a high masking ratio (80 %) can significantly improve the ability of the model to learn useful representations. • Multitask self-supervised learning is applied to basketball analytics. • Three tasks correspond to the three dimensions of basketball, from micro to macro. • The evaluation shows the necessity of each task for the model to understand basketball. • Both humans and AI rely on multilevel information to understand basketball.