AutoGaze: A Very Initial Exploration in A SAM2-based Pipeline for Automated Eye-Object Interaction Analysis in First-Person Videos

This paper presents a novel automated workflow for analyzing eye-tracking data in first-person videos. Our system uses the Segment Anything Model 2 (SAM2) to segment and track objects, correlating them with gaze information to provide a detailed understanding of visual attention and hand-object interactions. The pipeline outputs a structured JSON file containing rich information about gaze behavior, object interactions, and their temporal relationships. We demonstrate the system’s effectiveness in a life science operational context and compare its performance with a multimodal Large Language Model (LLM) for automated annotation. Our findings highlight the potential of combining advanced computer vision techniques with LLMs for comprehensive and scalable analysis of gaze behavior in real-world scenarios.