Probing Algebraic Structure in Neural Networks via Symmetric Cycle Closure

## Abstract Mechanistic interpretability of grokking [Power et al. 2022] typically relies on label accuracy or geometric proxies to claim that networks learn algebraic structure. We argue these indicators are insufficient and introduce **Symmetric Cycle Closure Count (Sym-CCC)**, a functional probe grounded in group axioms that tests bi-directional cancellation: a network respects group $G$ if both $\text{Net}(\text{Net}(a,b), b^{-1}) = a$ and $\text{Net}(a^{-1}, \text{Net}(a,b)) = b$ hold over $G \times G$. We derive the random baseline analytically ($1/|G|^2$, Lemma 3.1) and confirm it empirically on untrained networks. On dihedral group $D_{12}$ multiplication across 5 training seeds, we report three findings: 1. **Weight decay separates algebraic generalization from algebraic overfitting.** Without weight decay, networks reach 100% train accuracy with Train Sym-CCC = 53.9% ± 3.1% but Val Sym-CCC = 5.7% ± 1.8%, revealing local structural consistency without global coherence. With weight decay (0.1), both train and val Sym-CCC converge to 79–92%. 2. **Coordinate alignment accelerates but does not determine structure learning.** Under randomized index permutation, Sym-CCC reaches 72.6% ± 9.2% versus 79.4% ± 10.4% under contiguous indexing—a modest gap inconsistent with the "coordinate artifact" interpretation of single-seed measurements. 3. **A late-phase Float32 collapse is a precision-noise mechanism, not Softmax overflow.** Two mechanistically distinct interventions eliminate the collapse: increasing AdamW's $\epsilon$ from $10^{-8}$ to $10^{-4}$ (Val Sym-CCC at step 25k: 99.6% vs 90.4% standard), and switching to Float64 (100.0%). Direct gradient diagnostics rule out the Softmax overflow hypothesis (non-target gradient underflow remains 0.00% throughout). These results establish Sym-CCC as a direct functional probe for algebraic consistency, demonstrate compression as the operative force separating overfitting from generalization, and identify numerical precision as a determinant constraint in late-phase grokking dynamics. **Keywords:** grokking, mechanistic interpretability, group theory, weight decay, generalization, functional probes