A Pilot Mechanistic Study of Cataphoric Pronoun Resolution in GPT-2
Cataphora, a pronoun that precedes its antecedent (as in "Before she left, Maria locked the door"), requires maintaining an unresolved dependency rather than retrieving an already-established referent. Human sentence-processing research shows that comprehenders actively and persistently search for cataphoric antecedents, but to our knowledge no prior work examines this mechanistically in language models. We present a pilot study in GPT-2-small that combines behavioral measurement, activation patching, logit-lens analysis, linear probing and attention analysis. We find a large behavioral effect that is predominantly, though not entirely, attributable to syntactic structure; evidence that gender information is carried in the residual stream between the pronoun and its antecedent; heads and layers whose ordering suggests separate search and read-out stages; and an unexpected gender asymmetry in the model's default resolution behavior. We report these findings with attention to their limits.
Authors
- NEHAA SHRI MS (ORCID: https://orcid.org/0009-0001-9304-3039)
Institutions
- Vellore Institute of Technology University (IN)
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-11-16
- DOI
- https://doi.org/10.5281/zenodo.24283924
- Primary Topic
- Natural Language Processing Techniques
- Type
- preprint