Do Gauge-Geometric Components Improve Self-Attention? Causal Tests of a Learned Metric and Connection
Self-attention in standard Transformers uses query-key dot products to score tokens. We analyze whether learned geometry and a gauge connection can improve attention. The Gauge-Geometric Transformer (GGT) model scores attention using both dot-product and geodesic-inspired terms. It also applies a first-order gauge correction to value aggregation, and adds a commutator penalty as well as a curvature penalty on a separate probe metric during training. We analyze the model on WikiText-2, WikiText-103, and OpenWebText. The models in the five-seed WikiText-103 comparison have similar sizes (GGT has 0.25% more parameters than standard Transformer). However, GGT does not improve test perplexity over the standard Transformer. In addition, we analyze the contribution of the components added in GGT to the increase of cross-entropy. Removing the geodesic-inspired term from the attention score increases the cross-entropy by 0.072 nats on the trained Open WebText model, while removing the dot-product term increases it by 0.181 nats. Removing the gauge connection increases the cross-entropy by 0.050 nats. Reversing the gauge connection increases the cross-entropy by 0.314 nats and the perplexity by 36.8%. The attention metric field of the trained OpenWebText model has a token-dependent, anisotropic, low-dimensional structure and mostly negative sampled curvature. The endpoint-averaged attention-distance surrogate used in GGT violates the triangle inequality on 0.8% to 1.9% of sampled triples. The commutator penalty does not seem to increase non-commutativity. These geometric findings on one model need to be confirmed by running independent experiments.
Authors
- Gurpreet Singh (ORCID: https://orcid.org/0000-0002-6484-2704)
- Dr.Jasvir Singh
Institutions
- Punjabi University (IN)
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-11-21
- DOI
- https://doi.org/10.5281/zenodo.24413282
- Primary Topic
- Topic Modeling
- Type
- preprint