Algorithmic Induction via Structural Weight Transfer

# Engineering Algorithmic Structure in Neural Networks: A Materials Science Perspective **Author:** grisun0 --- ## Abstract This paper presents what I learned from attempting to induce Strassen matrix multiplication structure in neural networks, and why I now view this work as materials engineering rather than theory. I demonstrate through Strassen matrix multiplication that by controlling batch size, training duration, and regularization, I can induce discrete algorithmic structure that transfers zero-shot from 2x2 to 64x matrices. The two-phase protocol I present, training followed by sparsification and discretization, serves as empirical evidence. Under controlled conditions, 68% of runs crystallize into verifiable Strassen structure. The remaining 32% converge to local minima that generalize on test sets but fail structural verification. What I initially framed as a theory, claiming that gradient covariance geometry determines whether networks learn algorithms, did not hold up to scrutiny. Post-hoc analysis revealed that κ (the condition number I proposed) correlates with success but does not predict it prospectively. The hypothesis was backwards: successful models have κ≈1, but models with κ≈1 are not guaranteed to succeed. What remains valid is the engineering protocol itself. Here is what actually works: train with batch sizes in [24, 128], use weight decay ≥1e-4, run for 1000+ epochs, prune to 7 slots, round weights to integers. Do this, and you will induce Strassen structure with 68% probability. I now frame this work as materials engineering. We are not discovering fundamental laws. We are developing recipes for producing specific material properties in neural networks. The analogy is semiconductor manufacturing: we do not fully understand electron mobility at the quantum level, but we know exactly how to grow silicon crystals for chips. Similarly, I do not fully understand why batch size matters, but I know exactly how to tune training conditions to produce algorithmic structure. The system reveals extreme fragility: noise of magnitude 0.001 causes 100% discretization failure. This fragility has implications beyond my specific experiments. If a well-defined algorithm like Strassen requires such precise training conditions to emerge, what does this say about reproducibility in deep learning more broadly? The narrow basins containing algorithmic solutions may be far more common than we realize, and our inability to consistently reach them may explain many reproducibility failures in the field. --- ## 1. Introduction Neural networks trained on algorithmic tasks sometimes exhibit grokking: delayed generalization that occurs long after training loss has converged [1]. Prior work characterized this transition using local complexity measures [1] and connected it to superposition as lossy compression [2]. But a fundamental question remained unanswered: when a network groks, has it learned the algorithm, or has it found a local minimum that happens to generalize? This paper presents what I have learned from attempting to answer this question through Strassen matrix multiplication, and why I now view this work as materials engineering rather than theory. I set out to demonstrate that neural networks could learn genuine algorithms, not just convenient local minima. The test case was Strassen matrix multiplication, which has exact structure: 7 products with coefficients in {-1, 0, 1}. If a network learned Strassen, I could verify this by rounding weights to integers and checking if they matched the canonical structure. I developed a two-phase protocol. Phase 1: train a bilinear model with 8 slots on 2x2 multiplication. Phase 2: prune to 7 slots, discretize weights, and verify that the structure transfers to 64x64 matrices. I called this a theory. I claimed that the geometry of training trajectories determines whether algorithmic structure emerges. I proposed that gradient covariance, measured by κ, could predict which training runs would succeed. I was wrong about the prediction part. Post-hoc analysis showed that κ correlates with success but does not cause it, and cannot be used to predict outcomes from early-epoch measurements. What remains valid is the engineering protocol itself. When I follow the conditions I specify, Strassen structure emerges 68% of the time. This is a real result, reproducible, documented with 195 training runs. The batch size finding illustrates the engineering approach concretely. I observed that batch sizes in [24, 128] succeed while others fail. My initial hypothesis was hardware cache effects. I was wrong. Memory analysis showed even B=1024 fits comfortably in L3 cache (Appendix F). The batch size effect is real but unexplained. I do not have a theoretical explanation for why certain batch sizes favor convergence to discrete attractors. My contributions: 1. Engineering protocol: I provide a working recipe for inducing Strassen structure with 68% success rate. The conditions are specified, the success rate is documented, the verification framework is explicit. 2. Verification framework: I provide explicit criteria for distinguishing genuine algorithmic learning from local minima that generalize. 3. Honest limitations: I document what I tried, what worked, and what failed. The gradient covariance hypothesis is post-hoc correlation, not causal prediction. The batch size effect is unexplained. 4. Fragility implications: I discuss what the extreme sensitivity of algorithmic crystallization implies for reproducibility in deep learning. 5. Statistical validation: 195 training runs confirm that batch size significantly affects crystallization (F=15.34, p<0.0001, eta squared = 0.244). --- ## 2. Problem Setting I consider 2x2 matrix multiplication: C = A @ B A bilinear model learns tensors U, V, W such that: M_k = (U[k] . a) * (V[k] . b) c = W @ M where a, b, c are flattened 4-vectors. The central question is: Given a model with induced Strassen structure at 2x2, under what conditions can it be expanded to compute NxN matrix multiplication correctly without retraining? ### 2.1 Formal Definitions (Operational) The following definitions convert qualitative notions into measurable quantities: **Discretization operator Q(θ):** Post-hoc projection of coefficients to a discrete grid. In this work: rounding and clamping to {-1, 0, 1}. **Discretization margin δ(θ):** δ(θ) = ||θ - Q(θ)||_∞ A solution is "discretizable" if δ(θ) ≤ δ₀ for threshold δ₀ = 0.1 (weights within 0.1 of target integers). **Discrete success S(θ):** Binary event where S(θ) = 1 if Q(θ) matches the target structure (all 21 Strassen coefficients round correctly); S(θ) = 0 otherwise. This converts "crystallization" into a measurable order parameter. **Grokking (operational definition):** An interval of at least 100 epochs where training loss < 10⁻⁶ while test loss > 0.1, followed by an abrupt drop in test loss. **Control parameter:** Batch size B is the dominant control parameter. Other variables (epochs, weight decay, symmetric initialization) are treated as conditions or confounds. **Order parameter Φ(B):** Φ(B) = P[S(θ) = 1 | B] The probability of discrete success conditioned on batch size. Alternatively, E[δ(θ) | B] provides a continuous measure. **Gradient noise covariance:** For gradient gₜ = ∇_θ L(θₜ; Bₜ): Σₜ = Cov(gₜ | θₜ) σ²ₜ = Tr(Σₜ) / d, where d = dim(θ) **Normalized diffusion constant γₜ:** γₜ = (η/B) σ²ₜ The stabilized value γ₀ = lim_{t→∞} γₜ in the coherent regime characterizes the gradient noise geometry. **Critical batch size B_crit:** The minimum B such that γₜ stabilizes and Φ(B) shows a jump. Empirically observed in [24, 128], not thousands. **Fragility:** Quantified by P[S(Q(θ + ε)) = 1] with ε ~ N(0, σ²I). The paper reports 0% success for σ ≥ 0.001, indicating extremely narrow basins of attraction. --- ## 3. Methodology ### 3.1 The Two-Phase Protocol I use a two-phase protocol to induce and verify algorithmic structure. Phase 1, Training: I train a bilinear model with 8 slots on 2x2 matrix multiplication. The model learns tensors U, V, W such that C = W @ ((U @ a) * (V @ b)), where a and b are flattened input matrices. I use AdamW optimizer with weight decay at least 1e-4, batch sizes in [24, 128], and train for 1000+ epochs until grokking occurs. Phase 2, Sparsification and Discretization: After training, I prune to exactly 7 active slots based on importance scores (L2 norm of each slot). I then discretize all weights to integers in the set negative one, zero, one using rounding. Finally, I verify that the discretized coefficients produce correct matrix multiplication. Both phases are necessary. Phase 1 alone is not sufficient. In my early experiments, I ran only Phase 1 and observed 0% success. The model converged to solutions with 8 active slots and non-integer weights that did not match Strassen structure. Only after implementing Phase 2 with explicit sparsification did I achieve 68% success. This is not algorithm discovery. I am inducing a known structure through strong priors and explicit intervention. What is novel is the engineering protocol that makes this induction reliable and verifiable. Table: What is Engineered vs What Emerges | Feature | Engineered | Emergent | |---------|------------|----------| | Rank-7 constraint | Yes, via sparsification | No | | Integer coefficients | Yes, via discretization | No | | Convergence to discrete-compatible values | Partial | Partial | | Zero-shot transfer | No | Yes, when conditions met | Success rate without fallback: 68% (133/195 runs). Runs that fail Phase 2 are not counted as success. ### 3.2 Training Conditions for Phase 1 Batch size: Values in [24, 128] correlate with successful discretization. I initially hypothesized this was due to L3 cache effects. After computing memory requirements (model: 384 bytes, optimizer state