EyeLLM: Using Lookback Fixations to Enhance Human-LLM Alignment for Text Completion

Recent advances in LLMs offer new opportunities for supporting student writing, particularly through real-time, composition-level feedback.However, for such support to be effective, LLMs need to generate text completions that align with the writer's internal representation of their developing message, a representation that is often implicit and difficult to observe.This paper investigates the use of eyetracking data, specifically lookback fixations during pauses in text production, as a cue to this internal representation.Using eye movement data from students composing texts, we compare human-generated completions with LLM-generated completions based on prompts that either include or exclude words and sentences fixated during pauses.We find that incorporating lookback fixations enhances human-LLM alignment in generating text completions.These results provide empirical support for generating fixation-aware LLM feedback and lay the foundation for future educational tools that deliver real-time, composition-level feedback grounded in writers' attention and cognitive processes. 1