Do Gauge-Geometric Components Improve Self-Attention? Causal Tests of a Learned Metric and Connection

Self-attention in standard Transformers uses query-key dot products to score tokens. We analyze whether learned geometry and a gauge connection can improve attention. The Gauge-Geometric Transformer (GGT) model scores attention using both dot-product and geodesic-inspired terms. It also applies a first-order gauge correction to value aggregation, and adds a commutator penalty as well as a curvature penalty on a separate probe metric during training. We analyze the model on WikiText-2, WikiText-103, and OpenWebText. The models in the five-seed WikiText-103 comparison have similar sizes (GGT has 0.25% more parameters than standard Transformer). However, GGT does not improve test perplexity over the standard Transformer. In addition, we analyze the contribution of the components added in GGT to the increase of cross-entropy. Removing the geodesic-inspired term from the attention score increases the cross-entropy by 0.072 nats on the trained Open WebText model, while removing the dot-product term increases it by 0.181 nats. Removing the gauge connection increases the cross-entropy by 0.050 nats. Reversing the gauge connection increases the cross-entropy by 0.314 nats and the perplexity by 36.8%. The attention metric field of the trained OpenWebText model has a token-dependent, anisotropic, low-dimensional structure and mostly negative sampled curvature. The endpoint-averaged attention-distance surrogate used in GGT violates the triangle inequality on 0.8% to 1.9% of sampled triples. The commutator penalty does not seem to increase non-commutativity. These geometric findings on one model need to be confirmed by running independent experiments.

Authors

Institutions

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-21
DOI
https://doi.org/10.5281/zenodo.24413282
Primary Topic
Topic Modeling
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Do Gauge-Geometric Components Improve Self-Attention? Causal Tests of a Learned Metric and Connection

Gurpreet Singh, Dr.Jasvir Singh
Zenodo (CERN European Organization for Nuclear Research)
Topic Modeling
preprint

Do Gauge-Geometric Components Improve Self-Attention? Causal Tests of a Learned Metric and Connection

Gurpreet Singh, Dr.Jasvir Singh
preprint en

Abstract

Self-attention in standard Transformers uses query-key dot products to score tokens. We analyze whether learned geometry and a gauge connection can improve attention. The Gauge-Geometric Transformer (GGT) model scores attention using both dot-product and geodesic-inspired terms. It also applies a first-order gauge correction to value aggregation, and adds a commutator penalty as well as a curvature penalty on a separate probe metric during training. We analyze the model on WikiText-2, WikiText-103, and OpenWebText. The models in the five-seed WikiText-103 comparison have similar sizes (GGT has 0.25% more parameters than standard Transformer). However, GGT does not improve test perplexity over the standard Transformer. In addition, we analyze the contribution of the components added in GGT to the increase of cross-entropy. Removing the geodesic-inspired term from the attention score increases the cross-entropy by 0.072 nats on the trained Open WebText model, while removing the dot-product term increases it by 0.181 nats. Removing the gauge connection increases the cross-entropy by 0.050 nats. Reversing the gauge connection increases the cross-entropy by 0.314 nats and the perplexity by 36.8%. The attention metric field of the trained OpenWebText model has a token-dependent, anisotropic, low-dimensional structure and mostly negative sampled curvature. The endpoint-averaged attention-distance surrogate used in GGT violates the triangle inequality on 0.8% to 1.9% of sampled triples. The commutator penalty does not seem to increase non-commutativity. These geometric findings on one model need to be confirmed by running independent experiments.

Zenodo (CERN European Organization for Nuclear Research)
Punjabi University (IN)
Topic Modeling
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon
Do Gauge-Geometric Components Improve Self-Attention? Causal Tests of a Learned Metric and Connection — Gurpreet Singh, Dr.Jasvir Singh · Zenodo (CERN European Organization for Nuclear Research) (2026) | TGRS Research Map | TGRS