When Attention Repairs Itself: A Mechanistic Study of Backup Heads in GPT-2

A persistent ambiguity in mechanistic interpretability research concerns the causal origin of the Hydra Effect, the observed increase in attention allocated by downstream "backup" attention heads to a target token after the ablation of functionally dominant "primary" heads. The prevailing hypothesis posits that this spike may be, at least in part, a passive mathematical artifact: removing primary-head contributions collapses signal variance in the residual stream, which triggers LayerNorm scaling changes. These LayerNorm-induced changes can raise a target token’s attention probability when competing logits shift downward or shift less favorably. Disentangling this passive denominator effect from target-directed pre-softmax logit changes has remained methodologically elusive under standard causal tracing. We introduce the Frozen Attention Logit Intervention, a surgical counterfactual technique that isolates the dynamic, ablation-induced pre-softmax logit of a single target token while simultaneously freezing all non-target pre-softmax logits to their clean-run baseline values. This construction eliminates the confound of background denominator inflation, yielding a counterfactual attention probability that reflects target-logit attention contributions. We apply this method to GPT-2 Small on a cohort of 200 Indirect Object Identification (IOI) prompts, evaluating all 144 attention heads across 12 layers. Our principal finding is that in GPT-2 Small on a 200-example mixed-template IOI cohort, ablating L9H9 and L10H0 induces a robust L10H10 increase in IO-directed attention. This effect survives zero, mean, and resample ablation controls. Path-stable FALI/Shapley decomposition shows that the attention increase is dominated by the IO-target attention-logit contribution, while background-logit changes oppose the spike on average. QK decomposition suggests the IO-target score increase is primarily query-driven and largely cosine-alignment-driven. Output mediation shows that the L10H10 attention-pattern change causally contributes to IO-S logit recovery under this intervention, explaining about 43.6% of L10H10’s zero-ablation degradation by aggregate ratio of means. We keep this evidence separate from the LayerNorm-scale freezing diagnostic, which shows that under the Layer-10 clean-scale intervention, 24.3% of the observed L10H10 attention spike is removed (reducing L10H10's attention delta from +0.1893 to +0.1433). Together, these results qualify the role of LayerNorm surges while supporting the presence of target-directed compensatory pathways under upstream failures.