Look Back, Look Ahead: Coreference Signals in Pretrained Language Model Attention

We investigate whether pairwise attention features improve coreference readout beyond pooled span representations in frozen language models with up to 7B parameters. We train mention-ranking probes on hidden states, attention features, or both. Across 15 decoders from 7 families and 3 encoders, combining the two gives the highest OntoNotes scores with gold mentions; decoder gains over hidden states range from +1.5 to +2.5 CoNLL F1, and most of the gain is lost when attention features are shuffled across pairs. Without target-domain threshold calibration, attention-only probes transfer best to the pronoun benchmarks GAP and WinoBias, although calibration changes their ranking against combined probes. In three models, mean-ablating the heads that the attention probe ranks highest lowers the hidden-state readout even after the probe is retrained, with much smaller effects for heads sampled outside the top 64. A frozen 0.5B model comes within 1.1 F1 of our LoRA baseline. For end-to-end resolution, attention that a span receives from later tokens helps detect mentions belonging to non-singleton coreference chains, raising detection precision at a 90% recall target from 48.1 to 63.5 for Qwen2.5-0.5B. An extension to relation extraction with given entities also benefits from combining the two representations.