The Mechanism behind Understanding Anaphoric <i>One</i> and Noun Phrase Ellipsis: How Language Models Work

Abstract This study examines how language models access and recover linguistic information associated with the correct antecedents of two types of anaphoric elements: Anaphoric one and Noun Phrase Ellipsis (NPE), both of which require antecedent resolution for proper interpretation. In this study, we created manually annotated datasets (collectively referred to as AO-NPE datasets) from English Wikipedia, containing annotated instances of Anaphoric one and NPE. We assessed model performance by fine-tuning BERT and RoBERTa, and by evaluating GPT-4o mini, LLaMA 3.2, and Gemma-2 in a zero-shot setting. We conducted experiments using the forced-choice task and open-ended antecedent identification task on both humans and models with the same datasets. Notably, GPT-4o mini performed comparably to humans and outperformed other language models. However, results from both types of tasks revealed that language models performed better in identifying the antecedent for NPE than for Anaphoric one, a pattern opposite to that observed in human data. Linear distance and number agreement between antecedents and anaphoric elements also affected the performance, with language models more strongly influenced by these factors than humans. These findings suggest that the underlying mechanisms of anaphoric comprehension in language models may not fully align with those employed by humans: while humans flexibly integrate syntactic, semantic, and discourse information, LLMs rely more heavily on surface-level cues such as linear distance, number agreement, and syntactic parallelism. Additionally, model performance dropped significantly in the open-ended identification task compared to the forced-choice task, while human performance remained relatively stable. This further suggests potential limitations in anaphoric processing.