Physics-Informed Trajectory Constraints in Transformer Residual Streams

We investigate whether physics-informed trajectory constraints—soft penalties borrowed fromthe PINN framework—can be imposed on a transformer language model’s residual stream duringtraining without degrading language modeling performance. We attach a lightweight scalar readout head to the residual stream and penalize trajectories that violate specified dynamical laws: monotonicity (the readout can only increase) and exponential decay (the readout fadesbetween events). Across 66 training runs spanning two datasets (TinyStories, WikiText-103),two model scales (10.8M, 25M parameters), and four conditions (vanilla, multitask, monotone,decay), we find that trajectory constraints cost 0–1.9% perplexity, with the cost shrinkingalong both scale and training duration. Constrained models exhibit expanded representationaldimensionality (+20–37% PCA participation ratio), achieve target recoverability r 15 2 > 0.90across constraint types, and—most strikingly—preserve features that unconstrained trainingspontaneously develops and then degrades. Our results establish PINN-style trajectory constraintsas a viable, low-cost mechanism for imposing structural properties on transformer representationsat training time