The Mathematics of Weight-Space Programming

Weight-space programming means deliberately editing a neural network's numerical weights to implement a specified change in its computation or behavior. For a language model, an example is changing the answer to one prompt while preserving its responses on a specified set of other prompts. Our aim is to determine when such selective changes are possible, construct edits that satisfy explicit preservation constraints, and quantify their minimum cost under a stated metric. We develop this framework for the final feed-forward module of a transformer, linking linear compatibility and edit capacity to nonlinear normalization and preservation geometry. First, with positive-epsilon LayerNorm, we characterize attainable linear observations of output scores, including selected scores and differences between scores, and reduce the nearest feasible residual to a strictly convex scalar problem. This gives a globally minimum-cost single-target edit under fixed-key module-output protection. Second, for a fixed output direction, we derive exact intervals that preserve prediction margins, minimum edit budgets, and certificates of infeasibility. Under an offset-sign condition, a full-matrix formulation is convex; a robust extension covers specified activation regions. Controlled experiments evaluate 480 target–donor pairs on fifteen small transformers and 3,840 full-forward edits. With eight output scores and residual width 32, behavioral optimization reduces median edit norms by 33–40%. This gain uses directions invisible to the readout and vanishes for complete-score targets when the centered readout is injective; partial observations can retain freedom. A separate CounterFact study evaluates 128 requests on each of two pretrained GPT-2 checkpoints. Relative to a matched exact-moment control, finite interval constraints improve held-out neighbor retention by 21.1 and 19.4 percentage points at 100% target success, with lower paraphrase success and higher test loss. The studies test different theorems and are not pooled. Exact guarantees apply to specified keys, margins, or activation regions; broader preservation remains empirical. Our editors are explicit algorithms, not demonstrations of autonomous weight editing by a language model.