Mechanistic Convergence and Divergence Across LLM Instances: Axes of Correspondence for Generalizable Interpretability Research

Abstract Research on Large Language Models (LLMs) increasingly focuses on identifying mechanistic explanations for their behaviors, yet the field lacks clear principles for determining when (and how) findings from one model instance generalize to another. One solution is to identify LLM “reference classes” based on legible, top-down properties of models (e.g., their size or training regime); however, this does not guarantee that models share similar mechanisms. We propose a complementary approach rooted in mechanistic convergence. Crucially, this raises a fundamental epistemological challenge: what does it mean to assert that two models have the “same” mechanism? Drawing from work in neurophysiology, we propose five potential axes of correspondence along which mechanistic claims might generalize, including: functional (whether they satisfy the same functional criteria), developmental (whether they develop at similar points during pretraining), positional (whether they occupy similar absolute or relative positions), relational (whether they interact with other model components in similar ways), and configurational (whether they correspond to particular regions or structures in weight-space). These axes provide organizing principles to investigate and interpret putative mechanisms across models. We then validate this framework in three case studies analyzing 1-back attention heads, first-token heads, and induction heads across pretraining in random seeds of the Pythia models (14M, 70M, 160M, 410M), revealing striking developmental convergence (and some positional convergence) across seeds. Seeds of larger models systematically show earlier onsets, steeper slopes, and higher peaks of each attention type. We also demonstrate empirically that 1-back heads precede induction heads both positionally and temporally in all seeds, pointing to systematic inter-dependencies in the formation of more complex mechanisms. We then replicate and extend key findings in an expanded sample of 68 model instances (spanning 3 model families) differing in architecture and training data. We conclude by addressing possible objections, and argue that progress on the generalizability of mechanistic interpretability research will consist in mapping constitutive design properties of LLMs to their emergent behaviors and mechanisms.