Mobile gaze tracking calibration and interaction on handheld device

Gaze-based interaction offers a promising avenue for natural, hands-free engagement with handheld mobile devices. However, its practical deployment is hindered by the challenge of maintaining accuracy and usability in dynamic, real-world contexts where user and device motion change constantly. This motion disrupts the critical eye-camera spatial relationship and impacts the quality of camera data, rendering static calibration methods ineffective. This thesis addresses these challenges through a multi-faceted investigation into robust mobile gaze estimation and makes several key contributions. We first explore effective gaze-based interfaces for a mobile reading application. This involves a systematic evaluation of four distinct gaze interfaces including explicit dwell, smooth pursuit, gaze gesture (Eye-Swipe), and implicit gaze prediction under both sitting and walking conditions. User studies demonstrate that motion conditions can significantly impact the usability of gaze interfaces. Driven by the results, we conduct in-depth quantitative analysis, synchronizing IMU and vision data. This analysis empirically confirmed that head-to-screen distance, device orientation, and head movements are the primary drivers accounting for over 75% of gaze estimation error. Building on these findings, this thesis introduces MAC-Gaze, a novel motion-aware continual calibration framework. MAC-Gaze leverages on-device IMU sensors to intelligently detect motion that requires recalibration and employs a replay-based continual learning strategy to adapt the gaze model to new motion contexts while mitigating catastrophic forgetting. This framework achieves significant error reductions compared to baseline methods, lowering error by up to 19.9% on the RGBDGaze dataset (from 1.76cm to 1.41cm) and 31.7% on the more dynamic MotionGaze dataset (from 2.81cm to 1.92cm). Overall, this thesis advances the state-of-the-art by providing empirically validated interaction designs, a quantitative understanding of motion-caused errors, and a practical continual calibration solution, paving the way for more robust and reliable gaze-based interaction on mobile devices in everyday, dynamic environments.