EgoPoints: Advancing Point Tracking for Egocentric Videos

We introduce EgoPoints, a benchmark for point tracking in egocentric videos. We annotate 4.7K challenging tracks in egocentric sequences. Compared to the popular TAP-Vid-DAVIS [7] evaluation benchmark, we include 9x more points that go out-of-view and 59x more points that require re-identification (ReID) after returning to view. To measure the performance of models on these challenging points, we introduce evaluation metrics that specifically monitor tracking performance on points in-view, out-of-view, and points that require re-identification. We then propose a pipeline to create semi-real sequences, with automatic ground truth. We generate 11 K such sequences by combining dynamic Kubric [17] objects with scene points from EPIC Fields [38]. When fine-tuning point tracking methods on these sequences and evaluating on our annotated EgoPoints sequences, we improve Co-Tracker [22] across all metrics, including the tracking accuracy <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\delta^{\ast}_{avg}$</tex> by 2.7 percentage points and accuracy on ReID sequences (<tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$ReID\delta_{avg}$</tex>) by 2.4 points. We also improve <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\delta^{\ast}_{avg}$</tex> and <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$ReID\delta_{avg}$</tex> of PIPs++ [42] by 0.3 and 2.8 respectively.