See Where You Read: Reading Tracking via Eye Gaze Tracking and Large Language Model

Losing track of reading progress when switching lines can be frustrating. Eye gaze tracking technology offers a potential solution by highlighting read paragraphs, aiding users in avoiding wrong line switches. However, the gap between gaze tracking accuracy (2-3 cm) and text line spacing (3-5 mm) makes direct application impractical. Existing methods leverage the linear reading pattern but fail during jump reading. This paper presents a reading tracking and highlighting system that supports both linear and jump reading. The system leverages the large language model’s contextual perception capability in aiding reading tracking. A reading tracking domain-specific line-gaze alignment opportunity is also exploited to enable dynamic and frequent calibration of the gaze results. Controlled experiments demonstrate reliable linear reading tracking with performance comparable to that of the state-of-the-art linear reading tracking solution, while supporting jump reading tracking with 84% accuracy. Furthermore, real-world field tests with 24 volunteers demonstrated the system’s effectiveness in tracking and highlighting read paragraphs, improving reading efficiency, and enhancing user experience.