Does the GLU Value Branch Want to Be Nonlinear? A Mechanistic Study of Gradient Transparency in Gated Feed-Forward Networks

Gated feed-forward networks such as SwiGLU are standard in Transformer language models, but while prior work varied the gate nonlinearity of a gated linear unit (GLU), the value branch is almost always left linear. We ask whether that convention reflects an optimization role. Under exact parameter matching we train nine FFN variants at 124M scale on OpenWebText (the five claim-bearing ones with three seeds) and log branch-level gradient norms, sign statistics, dead units and activation outliers alongside validation perplexity. No nonlinear-value variant improves on SwiGLU; only Swish-ReLU is separated from it by more than seed noise. The clearest cost is stability: under the single, effectively constant learning rate all variants shared, 8 of 12 nonlinear-value runs drift or diverge late in training (Swish-ReLU in every seed) against none of the 7 linear-value and dense runs (Fisher's exact p = 0.013); whether a decayed schedule closes this gap is untested. At a common pre-divergence checkpoint a nonlinear value branch receives a smaller share of the gradient, mainly because the gate's gradient grows; the predicted sign collapse occurs at the value branch but is undone by the output projection; and squared-ReLU branches amplify activation outliers several-fold, a known low-precision hazard. Separately, branch-gradient statistics rise roughly a thousand iterations before validation perplexity does, flagging 7 of 8 destabilizing runs with no false alarms, against 5 of 8 for a training-loss monitor under the same rule. Under these settings the linear value branch is preferred for an optimization reason, not for lack of representational alternatives. Preprint. Not peer reviewed. Code and data are available from the corresponding author on request.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-16
DOI
https://doi.org/10.5281/zenodo.24270079
Primary Topic
Topic Modeling
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Does the GLU Value Branch Want to Be Nonlinear? A Mechanistic Study of Gradient Transparency in Gated Feed-Forward Networks

Khaled Shaheen, Faisal Shaheen
3 links
Zenodo (CERN European Organization for Nuclear Research)
Topic Modeling
preprint

Does the GLU Value Branch Want to Be Nonlinear? A Mechanistic Study of Gradient Transparency in Gated Feed-Forward Networks

Khaled Shaheen, Faisal Shaheen
preprint en

Abstract

Gated feed-forward networks such as SwiGLU are standard in Transformer language models, but while prior work varied the gate nonlinearity of a gated linear unit (GLU), the value branch is almost always left linear. We ask whether that convention reflects an optimization role. Under exact parameter matching we train nine FFN variants at 124M scale on OpenWebText (the five claim-bearing ones with three seeds) and log branch-level gradient norms, sign statistics, dead units and activation outliers alongside validation perplexity. No nonlinear-value variant improves on SwiGLU; only Swish-ReLU is separated from it by more than seed noise. The clearest cost is stability: under the single, effectively constant learning rate all variants shared, 8 of 12 nonlinear-value runs drift or diverge late in training (Swish-ReLU in every seed) against none of the 7 linear-value and dense runs (Fisher's exact p = 0.013); whether a decayed schedule closes this gap is untested. At a common pre-divergence checkpoint a nonlinear value branch receives a smaller share of the gradient, mainly because the gate's gradient grows; the predicted sign collapse occurs at the value branch but is undone by the output projection; and squared-ReLU branches amplify activation outliers several-fold, a known low-precision hazard. Separately, branch-gradient statistics rise roughly a thousand iterations before validation perplexity does, flagging 7 of 8 destabilizing runs with no false alarms, against 5 of 8 for a training-loss monitor under the same rule. Under these settings the linear value branch is preferred for an optimization reason, not for lack of representational alternatives. Preprint. Not peer reviewed. Code and data are available from the corresponding author on request.

Zenodo (CERN European Organization for Nuclear Research)
Topic Modeling
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon