Reading Between the Lines: How Eye-Tracking Data can Inform Reading Strategies for Large Language Models

Large language models (LLMs) have made significant advancements in natural language processing, yet they still face challenges in tasks requiring deep comprehension of structurally complex and visually rich documents. Human reading patterns, captured through eye-tracking, provide valuable insights into how meaning is extracted from text, particularly in Visually Rich Documents (VRDs). In this work, we propose a novel approach to integrating eye-tracking data into LLMs to enhance reading strategies that more closely reflect human cognitive processes. By mimicking human gaze paths when reading a VRD, we explore how these insights can improve LLMs’ comprehension abilities. Our experiments demonstrate that LLMs enhanced with human-like reading orders outperform baseline models in VRD understanding tasks. These findings suggest that incorporating human-like reading strategies can bridge the gap between machine and human understanding. Overall, our results indicate that integrating eye-tracking data significantly enhances LLM performance, paving the way for more human-like document comprehension in future AI systems.