Pruning Drift Is Local: Weight-Space Repair Without Gradients

Cut width from every layer of a trained transformer and its predictions drift. The standard remedies either retrain the surviving weights with gradients or reconstruct hidden states sequentially along the drifted prefix; this note tests a zero-gradient alternative in weight space, treating each pruned block as a local square whose operator defect is repaired directly. The repair follows a project-then-edit principle with no backpropagation: closed-form projection on clean layer inputs restores global subspace transport across the surviving width, and forward-only weight-space surgery adjusts individual readout directions where narrow token margins still flip. Two elementary propositions govern the regime: isotropic contraction of residual error can strictly widen a wrong token's logit lead whenever gauge-space error shrinks faster than logical readout error, and sequential prefix regression suffers errors-in-variables attenuation that shrinks layer gains on clean inputs. Measurements on a 42-layer Gemma 4 E4B model (15.99 GB in BF16) confirm both predictions under cryptographic prompt seals, runtime zero-gradient guards, and isolated cold-process evaluation. Physically slicing feed-forward width to 6,144 removes 16.52% of checkpoint weight bytes (13.35 GB, exact byte parity across all compressed variants). Uncompensated peak-protected slicing drops sealed accuracy from 23/24 to 19/24 and breaks all three multi-step arithmetic tests; local same-input projection recovers 21/24, restores all three arithmetic circuits, and matches the unpruned teacher's pass-fail behavior on 91.7% of unseen prompts. Six rounds of zero-gradient weight-space surgery close the remaining practice gap (11/12 to 12/12) and undo a calibration-induced character transposition on unseen copying while tying local projection at 21/24 on sealed pass-fail. In a 36-case corrected audit, local same-input projection (34/36) outperforms sequential live-prefix reconstruction (30/36) and sequential spectral weighting (29/36), while Stokes face billing isolates residual character corruption to a single mid-network block (Layer 23). Every claim in this note is either proved in elementary linear algebra or measured under sealed audit locks.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-15
DOI
https://doi.org/10.5281/zenodo.24241117
Primary Topic
Advanced Neural Network Applications
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Pruning Drift Is Local: Weight-Space Repair Without Gradients

Vincent Marquez
Zenodo (CERN European Organization for Nuclear Research)
Advanced Neural Network Applications
preprint

Pruning Drift Is Local: Weight-Space Repair Without Gradients

Vincent Marquez
preprint en

Abstract

Cut width from every layer of a trained transformer and its predictions drift. The standard remedies either retrain the surviving weights with gradients or reconstruct hidden states sequentially along the drifted prefix; this note tests a zero-gradient alternative in weight space, treating each pruned block as a local square whose operator defect is repaired directly. The repair follows a project-then-edit principle with no backpropagation: closed-form projection on clean layer inputs restores global subspace transport across the surviving width, and forward-only weight-space surgery adjusts individual readout directions where narrow token margins still flip. Two elementary propositions govern the regime: isotropic contraction of residual error can strictly widen a wrong token's logit lead whenever gauge-space error shrinks faster than logical readout error, and sequential prefix regression suffers errors-in-variables attenuation that shrinks layer gains on clean inputs. Measurements on a 42-layer Gemma 4 E4B model (15.99 GB in BF16) confirm both predictions under cryptographic prompt seals, runtime zero-gradient guards, and isolated cold-process evaluation. Physically slicing feed-forward width to 6,144 removes 16.52% of checkpoint weight bytes (13.35 GB, exact byte parity across all compressed variants). Uncompensated peak-protected slicing drops sealed accuracy from 23/24 to 19/24 and breaks all three multi-step arithmetic tests; local same-input projection recovers 21/24, restores all three arithmetic circuits, and matches the unpruned teacher's pass-fail behavior on 91.7% of unseen prompts. Six rounds of zero-gradient weight-space surgery close the remaining practice gap (11/12 to 12/12) and undo a calibration-induced character transposition on unseen copying while tying local projection at 21/24 on sealed pass-fail. In a 36-case corrected audit, local same-input projection (34/36) outperforms sequential live-prefix reconstruction (30/36) and sequential spectral weighting (29/36), while Stokes face billing isolates residual character corruption to a single mid-network block (Layer 23). Every claim in this note is either proved in elementary linear algebra or measured under sealed audit locks.

Zenodo (CERN European Organization for Nuclear Research)
Advanced Neural Network Applications
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon