Does the GLU Value Branch Want to Be Nonlinear? A Mechanistic Study of Gradient Transparency in Gated Feed-Forward Networks
Gated feed-forward networks such as SwiGLU are standard in Transformer language models, but while prior work varied the gate nonlinearity of a gated linear unit (GLU), the value branch is almost always left linear. We ask whether that convention reflects an optimization role. Under exact parameter matching we train nine FFN variants at 124M scale on OpenWebText (the five claim-bearing ones with three seeds) and log branch-level gradient norms, sign statistics, dead units and activation outliers alongside validation perplexity. No nonlinear-value variant improves on SwiGLU; only Swish-ReLU is separated from it by more than seed noise. The clearest cost is stability: under the single, effectively constant learning rate all variants shared, 8 of 12 nonlinear-value runs drift or diverge late in training (Swish-ReLU in every seed) against none of the 7 linear-value and dense runs (Fisher's exact p = 0.013); whether a decayed schedule closes this gap is untested. At a common pre-divergence checkpoint a nonlinear value branch receives a smaller share of the gradient, mainly because the gate's gradient grows; the predicted sign collapse occurs at the value branch but is undone by the output projection; and squared-ReLU branches amplify activation outliers several-fold, a known low-precision hazard. Separately, branch-gradient statistics rise roughly a thousand iterations before validation perplexity does, flagging 7 of 8 destabilizing runs with no false alarms, against 5 of 8 for a training-loss monitor under the same rule. Under these settings the linear value branch is preferred for an optimization reason, not for lack of representational alternatives. Preprint. Not peer reviewed. Code and data are available from the corresponding author on request.
Authors
- Khaled Shaheen
- Faisal Shaheen (ORCID: https://orcid.org/0009-0006-8905-6765)
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-11-16
- DOI
- https://doi.org/10.5281/zenodo.24270079
- Primary Topic
- Topic Modeling
- Type
- preprint