How does Face Detection Help Multi-person Tracking and Recognition in Metaverse 360 Degree Videos?

A new 360 degree video application has emerged for Metaverse, where a user is surrounded by several vertical-orientation videos playing different video streams. Each of these vertical-screen videos tries to focus on a given actor (e.g., a well-known dancer) in its video stream through multi-person tracking. However, conventional multi-tracking techniques fail to continuously track a known person.In this paper, we present a novel and effective method for multi-person tracking in such a 360 degree video application for Metaverse. Our approach decomposes the multi-person tracking task into several steps, proposing a top-down pipeline to address the problem. A key highlight of our work is the extraction of facial features, which enables the continuous tracking and recognition of individuals even when they are temporarily lost or obscured in the video stream. To achieve this goal, we employ a fused model of multi-person tracking and face recognition. Rather than simply integrating these two models, we develop a comprehensive evaluation framework to match tracked individuals based on various criteria, such as facial similarity, pose similarity, Intersection over Union (IoU) scores, and prioritize these matches accordingly. We implement a universal method for dynamically calculating thresholds to adapt across different video datasets. Evaluation results show that our proposed approaches advance the state of the art in both general multi-person tracking and single-person tracking, as well as the new task of tracking a specific individual using a reference image in a video stream.