Leveraging Temporal Gaze Patterns for Intelligent Segmentation in Mobile Eye-Tracking

Recent advancements in deep learning models for object segmentation have significantly enhanced the analysis of mobile eye-tracking data, enabling automatic dataset annotation. However, many of these models rely solely on raw gaze and fixation points to indicate annotation targets on individual video frames, thereby neglecting the valuable temporal contexts provided by gaze features. In this paper, we introduce a pipeline that integrates temporal gaze features into the SAM 2 model to improve the segmentation accuracy of automatic annotations for mobile eye-tracking data. Specifically, we investigate different types of encoding temporal gaze information, used as prompts, to help the model learn gaze patterns at various levels of object granularity and to account for inaccuracies in gaze location introduced by mobile scenarios. Our experiments show that adding temporal context largely benefits the segmentation performance. In addition, our proposed fusion strategy allows us to combine different types of prompts and shows promising results.