Dynamic Digital Twins via a Fusion of Radiance Fields and Camera Feeds

We describe a complete methodology for constructing a dynamic digital twin of a sporting event. We focus on ice hockey and show how we can efficiently render the players from an arbitrary camera view by fusing fixed camera feeds of the action with a radiance field representation of the arena. We first compare algorithms for training the radiance field, and describe methods for the detection and segmentation of players from the fixed camera feeds. Once we have the player representations we calculate spatio-temporal correlations to track each player over time. From these trajectories we show how to pick a representation for every player at each time instant, and then superimpose the players on an image of the arena rendered from the radiance field.