Algorithmic Induction via Structural Weight Transfer

# Engineering Algorithmic Structure in Neural Networks: From a Materials Science Perspective to Algorithmic Thermodynamics of Deep Learning **Author:** Iscomeback, Gris ( grisun0 ) --- ## Abstract This paper presents what I learned from attempting to induce Strassen matrix multiplication structure in neural networks, and why I now view this work as materials engineering rather than theory. I demonstrate through Strassen matrix multiplication that by controlling batch size, training duration, and regularization, I can induce discrete algorithmic structure that transfers zero-shot from 2x2 to 64x matrices. The two-phase protocol I present, training followed by sparsification and discretization, serves as empirical evidence. Under controlled conditions, 68% of runs crystallize into verifiable Strassen structure. The remaining 32% converge to local minima that generalize on test sets but fail structural verification. What I initially framed as a theory, claiming that gradient covariance geometry determines whether networks learn algorithms, did not hold up to scrutiny. Post-hoc analysis revealed that κ (the condition number I proposed) correlates with success but does not predict it prospectively. The hypothesis was backwards: successful models have κ≈1, but models with κ≈1 are not guaranteed to succeed. Following reviewer feedback, I now have stronger evidence for κ as a predictive metric. Across 20 balanced runs with varied hyperparameters, κ achieves perfect separation between grokked and non-grokked outcomes (AUC = 1.000, 95% CI [1.000, 1.000]) on the validation set of 20 runs. While this indicates strong predictive power, the interval is degenerate because no overlap exists between classes. Future work should test generalization to unseen hyperparameter regimes. Additionally, κ prospectively separates grokked vs. non-grokked runs (N=60, AUC=1.000) within tested hyperparameter ranges, confirming that the metric reliably predicts outcomes before training completes. Local Complexity drops to zero exactly at the grokking transition (Figure 6), confirming it captures the phase change. The discrete basin remains stable under iterative pruning up to 50% sparsity, after which the solution collapses. The 60-run hyperparameter sweep provides conclusive validation. When I varied batch size from 8 to 256 and weight decay from 1e-5 to 1e-2, κ perfectly separated successful from failed runs. Every run that grokked showed κ = 1.000. Every run that failed showed κ = 999999. The AUC reached 1.000 with 95% CI [1.000, 1.000]. These results are the most definitive evidence I have that κ captures something real about training dynamics. What remains valid is the engineering protocol itself. Here is what actually works: train with batch sizes in [24, 128], use weight decay ≥1e-4, run for 1000+ epochs, prune to 7 slots, round weights to integers. Do this, and you will induce Strassen structure with 68% probability. I used to call this work “materials engineering” because I could not measure heat. Now I can. I ran 245 training runs, logged every gradient, and treated each checkpoint as a micro-state. The numbers gave me temperature, entropy, and heat capacity without metaphor. The recipe is still the same—batch size 32, weight decay 1e-4, 1000 epochs, prune to seven slots, round—but I no longer sell it as kitchen wisdom. It is a reproducible thermodynamic protocol that places a discrete algorithm at a predictable point in phase space. κ, the condition number of the gradient covariance matrix, acts as an order parameter: κ = 1.000 exactly when the system is in the crystal phase; κ = 999999 otherwise. Across sixty hyper-parameter configurations the separation is perfect (AUC = 1.000, 95 % CI [1.000, 1.000]). The confidence interval is degenerate because the two distributions do not overlap. Local Complexity drops from 442 to 0 at the grokking transition, confirming a first-order phase change. The crystal basin is stable under pruning up to 50 % sparsity and shatters at 55 %, giving a measurable yield stress. These are not literary devices; they are values extracted from logs. I write this note to record what the machine told me before I forget the difference between what I hoped and what I measured. **Phase imaging in the materials sense.** Figures in this work serve as experimental visualizations of microstructural properties: Figure 4 shows weight distribution evolution (microstructure), Figure 7 shows batch size effect (phase boundary), Figure 8 shows the complete phase diagram (phase map), Figure 5 shows grokking dynamics (temporal phase transition), and Appendix E shows noise perturbation results (basin width measurement). These images characterize the material properties of trained networks without claiming thermodynamic equivalence. The system reveals extreme fragility: noise of magnitude 0.001 causes 100% discretization failure when applied post-training. However, I now have evidence that the discrete basin is stable under pruning up to 50% sparsity. This fragility has implications beyond my specific experiments. If a well-defined algorithm like Strassen requires such precise training conditions to emerge, what does this say about reproducibility in deep learning more broadly? The narrow basins containing algorithmic solutions may be far more common than we realize, and our inability to consistently reach them may explain many reproducibility failures in the field. --- ## 1. Introduction Neural networks trained on algorithmic tasks sometimes exhibit grokking: delayed generalization that occurs long after training loss has converged [1]. Prior work characterized this transition using local complexity measures [1] and connected it to superposition as lossy compression [2]. But a fundamental question remained unanswered: when a network groks, has it learned the algorithm, or has it found a local minimum that happens to generalize? This paper presents what I have learned from attempting to answer this question through Strassen matrix multiplication, and why I now view this work as materials engineering rather than theory. I set out to demonstrate that neural networks could learn genuine algorithms, not just convenient local minima. The test case was Strassen matrix multiplication, which has exact structure: 7 products with coefficients in {-1, 0, 1}. If a network learned Strassen, I could verify this by rounding weights to integers and checking if they matched the canonical structure. I developed a two-phase protocol. Phase 1: train a bilinear model with 8 slots on 2x2 multiplication. Phase 2: prune to 7 slots, discretize weights, and verify that the structure transfers to 64x64 matrices. I called this a theory. I claimed that the geometry of training trajectories determines whether algorithmic structure emerges. I proposed that gradient covariance, measured by κ, could predict which training runs would succeed. I was wrong about the prediction part. Post-hoc analysis showed that κ correlates with success but does not cause it, and cannot be used to predict outcomes from early-epoch measurements. However, following reviewer-requested validation experiments, I now have prospective evidence that κ achieves perfect separation (AUC = 1.000, 95% CI [1.000, 1.000]) on the validation set of 20 runs. While this indicates strong predictive power, the interval is degenerate because no overlap exists between classes. Future work should test generalization to unseen hyperparameter regimes. This validates κ as a prospective prediction metric. What remains valid is the engineering protocol itself. When I follow the conditions I specify, Strassen structure emerges 68% of the time. This is a real result, reproducible, documented with 195 training runs. Without pruning, 0% of runs converge to Strassen structure (N=195), confirming that explicit sparsification is essential for algorithmic induction. The batch size finding illustrates the engineering approach concretely. I observed that batch sizes in [24, 128] succeed while others fail. My initial hypothesis was hardware cache effects. I was wrong. Memory analysis showed even B=1024 fits comfortably in L3 cache (Appendix F). The batch size effect is real but unexplained. I do not have a theoretical explanation for why certain batch sizes favor convergence to discrete attractors. This work presents Strassen matrix multiplication as a primary case study within a broader research program on algorithmic induction. The methods, metrics, and engineering protocols developed here are designed to extend to other algorithmic structures, including parity tasks, wave equations, and orbital dynamics. The broader program investigates whether the principles governing Strassen induction generalize across domains, with this paper providing the first systematic validation of the κ metric and pruning protocol. I wanted to know whether a neural network can learn Strassen multiplication instead of merely generalising on the test set. The only way I trust is to force the weights on to the exact integer coefficients that Strassen published. If the rounded model still multiplies matrices correctly at every scale, the algorithm is inside. Otherwise I have found a convenient minimum that happens to work on the data I fed it. The experiment is simple in principle: train, prune, round, verify. The difficulty is reaching the narrow region in weight space where rounding is harmless. I ran 245 full training trajectories and recorded every gradient, every eigenvalue of the covariance matrix, and every distance to the nearest integer lattice. Treating the final weights as micro-states gives me a partition function, an entropy, and a temperature. The numbers say there are two phases: glass (δ ≈ 0.49) and crystal (δ = 0). The transition is sharp; no checkpoint lives between them. κ is the control knob: set κ = 1 and you are in the crystal; any other value keeps you in the glass. I did