Algorithmic Induction via Structural Weight Transfer

# Trajectory Geometry Determines Algorithmic Structure: A Paradigm Shift in Neural Network Training **Author:** grisun0 --- ## Abstract The central claim of this work is a philosophical shift in how we understand neural network learning: the geometry of training trajectories matters more than the properties of final solutions. Algorithmic structure does not passively emerge from optimization. It is actively constructed through precise manipulation of gradient dynamics. I demonstrate this through Strassen matrix multiplication. By controlling batch size, training duration, and regularization, I induce discrete algorithmic structure that transfers zero-shot from 2x2 to 64x64 matrices. The two-phase protocol I present, training followed by sparsification and discretization, serves as empirical evidence for this claim. Under controlled conditions, 68% of runs crystallize into verifiable Strassen structure. The remaining 32% converge to local minima that generalize on test sets but fail structural verification. This distinction is the core insight. Previous grokking studies reported delayed generalization but could not determine whether networks learned genuine algorithms or found convenient local minima. I provide the verification framework to answer this question. The system reveals extreme fragility: noise of magnitude 0.001 causes 100% discretization failure. This fragility has implications beyond my specific experiments. If a well-defined algorithm like Strassen requires such precise training conditions to emerge, what does this say about reproducibility in deep learning more broadly? The narrow basins containing algorithmic solutions may be far more common than we realize, and our inability to consistently reach them may explain many reproducibility failures in the field. The batch size finding illustrates the paradigm shift concretely. I observed that batch sizes in [24, 128] succeed while others fail. My initial hypothesis, hardware cache effects, was wrong. Memory analysis showed even B=1024 fits in L3 cache. The true explanation lies in gradient covariance geometry: certain batch sizes create trajectories that favor convergence to discrete attractors. The solution properties are identical across batch sizes that reach them. What differs is the trajectory geometry that determines whether we reach them at all. --- ## 1. Introduction Neural networks trained on algorithmic tasks sometimes exhibit grokking: delayed generalization that occurs long after training loss has converged [1]. Prior work characterized this transition using local complexity measures [1] and connected it to superposition as lossy compression [2]. But a fundamental question remained unanswered: when a network groks, has it learned the algorithm, or has it found a local minimum that happens to generalize? This paper argues that the answer depends not on the solution found, but on the trajectory taken to reach it. The geometry of training dynamics determines whether a network crystallizes into genuine algorithmic structure or settles into a convenient local minimum. This is a paradigm shift: from analyzing solutions to analyzing trajectories. I demonstrate this through Strassen matrix multiplication. The algorithm has discrete structure: 7 multiplications with coefficients in the set negative one, zero, one. A network that truly learns Strassen should produce weights that discretize to these exact values and transfer to larger matrices without retraining. A network that merely finds a good local minimum will fail this verification. My two-phase protocol provides the empirical evidence: Phase 1, Training: Train a bilinear model with 8 slots on 2x2 matrix multiplication. Use batch sizes in [24, 128], weight decay at least 1e-4, and train for 1000+ epochs until grokking occurs. Phase 2, Verification: Prune to 7 active slots, discretize weights to integers, and verify that the structure computes correct matrix multiplication at scales from 2x2 to 64x64. Under these conditions, 68% of runs crystallize into verifiable Strassen structure. The remaining 32% converge to solutions that achieve low test loss but fail structural verification. These are the local minima that previous grokking studies could not distinguish from genuine learning. The system reveals extreme fragility. Adding noise of magnitude 0.001 to weights before discretization causes 100% failure. This has implications beyond my experiments. If an algorithm as well-defined as Strassen requires such precise conditions to emerge, reproducibility failures in deep learning may often reflect trajectories that missed narrow basins rather than fundamental limitations of architectures or data. The batch size investigation illustrates the paradigm shift concretely. I observed that B in [24, 128] succeeds while other values fail. My initial hypothesis was hardware cache effects. I was wrong. Memory analysis showed even B=1024 fits comfortably in L3 cache (Appendix F). The true explanation lies in gradient covariance geometry. Certain batch sizes create trajectories where gradient noise balances exploration and stability, favoring convergence to discrete attractors. The destination is the same. What differs is whether the trajectory reaches it. My contributions: 1. Paradigm claim: I argue that trajectory geometry determines algorithmic structure. The two-phase protocol is the empirical evidence, not the contribution itself. 2. Verification framework: I provide explicit criteria for distinguishing genuine algorithmic learning from local minima that generalize. 3. Batch size investigation: I document the scientific method in action, from initial observation through rejected hypothesis to proposed mechanism. 4. Fragility implications: I discuss what the extreme sensitivity of algorithmic crystallization implies for reproducibility in deep learning. 5. Statistical validation: 195 training runs confirm that batch size significantly affects crystallization (F=15.34, p<0.0001, eta squared = 0.244). --- ## 2. Problem Setting I consider 2x2 matrix multiplication: C = A @ B A bilinear model learns tensors U, V, W such that: M_k = (U[k] . a) * (V[k] . b) c = W @ M where a, b, c are flattened 4-vectors. The central question is: Given a model with induced Strassen structure at 2x2, under what conditions can it be expanded to compute NxN matrix multiplication correctly without retraining? ### 2.1 Formal Definitions (Operational) The following definitions convert qualitative notions into measurable quantities: **Discretization operator Q(θ):** Post-hoc projection of coefficients to a discrete grid. In this work: rounding and clamping to {-1, 0, 1}. **Discretization margin δ(θ):** δ(θ) = ||θ - Q(θ)||_∞ A solution is "discretizable" if δ(θ) ≤ δ₀ for threshold δ₀ = 0.1 (weights within 0.1 of target integers). **Discrete success S(θ):** Binary event where S(θ) = 1 if Q(θ) matches the target structure (all 21 Strassen coefficients round correctly); S(θ) = 0 otherwise. This converts "crystallization" into a measurable order parameter. **Grokking (operational definition):** An interval of at least 100 epochs where training loss < 10⁻⁶ while test loss > 0.1, followed by an abrupt drop in test loss. **Control parameter:** Batch size B is the dominant control parameter. Other variables (epochs, weight decay, symmetric initialization) are treated as conditions or confounds. **Order parameter Φ(B):** Φ(B) = P[S(θ) = 1 | B] The probability of discrete success conditioned on batch size. Alternatively, E[δ(θ) | B] provides a continuous measure. **Gradient noise covariance:** For gradient gₜ = ∇_θ L(θₜ; Bₜ): Σₜ = Cov(gₜ | θₜ) σ²ₜ = Tr(Σₜ) / d, where d = dim(θ) **Normalized diffusion constant γₜ:** γₜ = (η/B) σ²ₜ The stabilized value γ₀ = lim_{t→∞} γₜ in the coherent regime characterizes the gradient noise geometry. **Critical batch size B_crit:** The minimum B such that γₜ stabilizes and Φ(B) shows a jump. Empirically observed in [24, 128], not thousands. **Fragility:** Quantified by P[S(Q(θ + ε)) = 1] with ε ~ N(0, σ²I). The paper reports 0% success for σ ≥ 0.001, indicating extremely narrow basins of attraction. --- ## 3. Methodology ### 3.1 The Two-Phase Protocol I use a two-phase protocol to induce and verify algorithmic structure. Phase 1, Training: I train a bilinear model with 8 slots on 2x2 matrix multiplication. The model learns tensors U, V, W such that C = W @ ((U @ a) * (V @ b)), where a and b are flattened input matrices. I use AdamW optimizer with weight decay at least 1e-4, batch sizes in [24, 128], and train for 1000+ epochs until grokking occurs. Phase 2, Sparsification and Discretization: After training, I prune to exactly 7 active slots based on importance scores (L2 norm of each slot). I then discretize all weights to integers in the set negative one, zero, one using rounding. Finally, I verify that the discretized coefficients produce correct matrix multiplication. Both phases are necessary. Phase 1 alone is not sufficient. In my early experiments, I ran only Phase 1 and observed 0% success. The model converged to solutions with 8 active slots and non-integer weights that did not match Strassen structure. Only after implementing Phase 2 with explicit sparsification did I achieve 68% success. This is not algorithm discovery. I am inducing a known structure through strong priors and explicit intervention. What is novel is the engineering protocol that makes this induction reliable and verifiable. Table: What is Engineered vs What Emerges | Feature | Engineered | Emergent | |---------|------------|----------| | Rank-7 constraint | Yes, via sparsification | No | | Integer coefficients | Yes, via discretization | No | | Convergence to discrete-compatible values | Partial | Partial | | Zero-shot transfer | No | Yes, when conditions met | Success rate without fallback: 68% (133/195