Pruning Drift Is Local: Weight-Space Repair Without Gradients
Cut width from every layer of a trained transformer and its predictions drift. The standard remedies either retrain the surviving weights with gradients or reconstruct hidden states sequentially along the drifted prefix; this note tests a zero-gradient alternative in weight space, treating each pruned block as a local square whose operator defect is repaired directly. The repair follows a project-then-edit principle with no backpropagation: closed-form projection on clean layer inputs restores global subspace transport across the surviving width, and forward-only weight-space surgery adjusts individual readout directions where narrow token margins still flip. Two elementary propositions govern the regime: isotropic contraction of residual error can strictly widen a wrong token's logit lead whenever gauge-space error shrinks faster than logical readout error, and sequential prefix regression suffers errors-in-variables attenuation that shrinks layer gains on clean inputs. Measurements on a 42-layer Gemma 4 E4B model (15.99 GB in BF16) confirm both predictions under cryptographic prompt seals, runtime zero-gradient guards, and isolated cold-process evaluation. Physically slicing feed-forward width to 6,144 removes 16.52% of checkpoint weight bytes (13.35 GB, exact byte parity across all compressed variants). Uncompensated peak-protected slicing drops sealed accuracy from 23/24 to 19/24 and breaks all three multi-step arithmetic tests; local same-input projection recovers 21/24, restores all three arithmetic circuits, and matches the unpruned teacher's pass-fail behavior on 91.7% of unseen prompts. Six rounds of zero-gradient weight-space surgery close the remaining practice gap (11/12 to 12/12) and undo a calibration-induced character transposition on unseen copying while tying local projection at 21/24 on sealed pass-fail. In a 36-case corrected audit, local same-input projection (34/36) outperforms sequential live-prefix reconstruction (30/36) and sequential spectral weighting (29/36), while Stokes face billing isolates residual character corruption to a single mid-network block (Layer 23). Every claim in this note is either proved in elementary linear algebra or measured under sealed audit locks.
Authors
- Vincent Marquez
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-11-15
- DOI
- https://doi.org/10.5281/zenodo.24241117
- Primary Topic
- Advanced Neural Network Applications
- Type
- preprint