Algorithmic Induction via Structural Weight Transfer

# Engineering Generalization: Conditions for Stable Algorithmic Transfer in Neural Networks **Author:** grisun0 --- ## Abstract This paper establishes a set of empirical engineering conditions that are necessary for stable algorithmic transfer in neural networks: the property that a trained model can be expanded to larger input dimensions without retraining while preserving correct computation. I demonstrate this using bilinear models trained on 2x2 matrix multiplication with Strassen-structured inductive bias. I do not claim that networks discover algorithms from scratch. Instead, I induce a known structure (rank-7 tensor decomposition) through architectural constraints and post-hoc discretization, then identify the precise conditions under which this induced structure transfers to larger problem sizes. Structural transfer succeeds when engineering conditions are met (68% of runs); it fails otherwise (32% of runs). When successful, the induced structure generalizes to 4x4, 8x8, 16x16, 32x32, and 64x64 matrices. The contribution is an engineering guide for structural transfer. I establish that batch sizes in the range [24, 128], training duration of 1000+ epochs, and weight decay regularization (>= 1e-4) are necessary conditions for stable discretization and zero-shot scaling. Under these conditions, the induced Strassen implementation achieves 1.95x speedup over single-threaded OpenBLAS at N=8192. The system exhibits extreme fragility to noise (0% success with sigma >= 0.001), which underscores why precise engineering of training conditions is essential. Statistical validation across 195 training runs confirms that batch size significantly affects convergence quality (F=15.34, p<0.0001). > **Core thesis:** Stable algorithmic transfer is a property of training trajectories constrained by gradient noise geometry, not of learned solutions. This work represents a step toward engineering neural networks whose large-scale behavior is predictable from small-scale training. --- ## 1. Introduction Neural networks trained on algorithmic tasks sometimes exhibit grokking: delayed generalization that occurs long after training loss has converged [1]. Prior work by Humayun et al. [1] characterized this transition using local complexity measures, and Bereska et al. [2] connected it to superposition as lossy compression. I address a fundamental question: under what conditions can an induced algorithmic structure be engineered to transfer reliably to larger problem instances? This is not a claim about algorithm discovery. It is a question about engineering generalization. The central claim of this work is: > **Stable algorithmic transfer is not a property of solutions, but of training trajectories constrained by gradient noise geometry.** There exists a batch size regime where the gradient covariance induces trajectories that collapse to stable discrete representations, enabling zero-shot structural transfer. Strassen matrix multiplication serves as the experimental microscope for observing this phenomenon; the underlying principle is general. The system I study is inherently fragile. Adding Gaussian noise with sigma as small as 0.001 to trained weights causes 100% failure in discretization. This fragility is not a weakness of my method; it is a fundamental property of the problem. Precisely because the system is so sensitive, a precise engineering guide is essential. This paper provides that guide. My experimental setup uses explicit inductive bias: 1. The model architecture uses rank-8 tensor decomposition with a target of 7 active slots (matching Strassen).2. After training, weights are discretized to {-1, 0, 1} via rounding.3. If verification fails, the system falls back to canonical Strassen coefficients. Given this methodology, I establish the engineering conditions under which the induced structure remains stable during expansion without retraining. My contributions: 1. Engineering conditions: I establish that batch sizes in [24, 128], training duration of 1000+ epochs, and weight decay >= 1e-4 are necessary conditions for stable structural transfer. Success rate: 68% without fallback. 2. Batch size as critical parameter: I identify batch size as the dominant factor (eta^2 = 0.244, explaining 24% of variance), and propose gradient covariance dynamics as the underlying mechanism. 3. Uniqueness of expansion operator: I verify experimentally that slot ordering is essential. Permuting slots breaks correctness (mean error 74%), confirming the expansion operator T is unique for a given coefficient ordering. 4. Statistical validation: I present experimental validation with N=195 observations confirming significant effects of batch size on convergence (F=15.34, p<0.0001). --- ## 2. Problem Setting I consider 2x2 matrix multiplication: C = A @ B A bilinear model learns tensors U, V, W such that: M_k = (U[k] . a) * (V[k] . b) c = W @ M where a, b, c are flattened 4-vectors. The central question is: Given a model with induced Strassen structure at 2x2, under what conditions can it be expanded to compute NxN matrix multiplication correctly without retraining? --- ## 3. Methodology ### 3.1 Inductive Bias I am explicit about the inductive bias in my approach: 1. Architecture: The model uses 8 slots, with a target of 7 active slots (matching Strassen's rank). 2. Sparsification: After training, I prune to exactly 7 slots based on importance scores. 3. Discretization: Weights are rounded to {-1, 0, 1} using torch.round().clamp(-1, 1). This is post-hoc intervention, not emergent behavior. 4. Fallback: If verification fails, canonical Strassen coefficients are used (32% of runs). This is not algorithm discovery. It is structured optimization with strong priors. Table: Engineered vs Emergent Features | Feature | Engineered | Emergent ||---------|------------|----------|| Rank-7 constraint | Yes (architectural prior) | No (engineered) || Values {-1, 0, 1} | Yes (post-hoc rounding) | No (engineered) || Convergence to discrete | Partial (training dynamics) | Partial || Benchmark performance | No | Yes || Zero-shot transfer | No | Yes (when conditions met) | Success rate without fallback: 68% (133/195 runs). CV of discretization error: 1.2%. ### 3.2 Training Conditions I investigate how training parameters affect convergence: Batch size: Values in [24, 128] correlate with successful discretization. Correction: I initially hypothesized this was due to L3 cache coherence. After computing memory requirements (model: 384 bytes, optimizer state: 768 bytes, per-sample: 320 bytes), I found that even B=1024 fits comfortably in L3 cache on all tested hardware. The batch size effect is therefore due to training dynamics (gradient noise, learning rate coupling), not hardware constraints. I do not yet have a theoretical explanation for why [24, 128] works best. Training duration: Extended training (1000+ epochs) is required for weights to approach values amenable to discretization. Optimizer: AdamW with weight decay >= 1e-4 produces better results than pure Adam. ### 3.3 Verification Protocol After discretization, I verify: 1. Correctness: C_model matches C_true within floating-point tolerance (relative error < 1e-5)2. Expansion: The same coefficients work for 4x4, 8x8, 16x16, 32x32, 64x64 Discretization success is defined as: all 21 weight values (7 slots x 3 tensors) round to the correct Strassen coefficient. Partial success is not counted. ### 3.4 Discretization Fragility: The Reason Engineering Matters I tested noise stability by adding Gaussian noise (sigma in {0.001, 0.01, 0.1}) to weights before discretization. Success rate dropped to 0% for all noise levels tested (100 trials each). This extreme fragility is not a limitation of the method; it is the fundamental justification for why precise engineering of training conditions is essential. The algorithmic structure exists in a narrow basin of attraction. Small perturbations destroy discretization completely. This property underscores the importance of the engineering guide established in this work: without precise control of batch size, training duration, and regularization, the system cannot reliably reach the discrete attractor. The fragility transforms from apparent weakness to core insight: navigating to stable algorithmic structure requires exact engineering, and this paper provides the necessary conditions for that navigation. --- ## 4. Convergence Conditions ### 4.1 Empirically Validated Proposition Proposition 4.1 (Conditions for Successful Discretization) Note: These are empirical observations, not derived theorems. I observe that discretization succeeds (weights round to correct Strassen coefficients) when: (A1) Batch size B is in [24, 128]. (A2) Training continues for at least 500 epochs with grokking dynamics observed. I define grokking as: training loss < 1e-6 while test loss remains > 0.1 for at least 100 epochs, followed by sudden test loss drop (see Appendix D, Figure 5). (A3) Weight decay is applied (>= 1e-4 for AdamW). (A4) The model uses symmetric initialization for U and V tensors. When these conditions are met, weights typically approach values within 0.1 of {-1, 0, 1}, making discretization reliable. The metric is L-infinity: max(|w - round(w)|) < 0.1 for all weights. When conditions are not met, the fallback to canonical coefficients is triggered automatically by the verification step. --- ## 5. Algebraic Formalization: Theory and Verification **Note:** This section provides a descriptive framework, not a predictive theory. The formal definitions offer a language for describing the phenomena observed in the experiments; they are not claimed as proven theorems. No novel mathematical results are introduced here. Readers primarily interested in the empirical findings may proceed to Section 6. This section presents the general theory developed in my prior work, then describes how the Strassen