Quaternion-Tucker ZigZag Networks: Halving Transformer Backbone Parameters via Shared Hypercomplex Tensor Cores and Braided Top-Down Routing

Standard Transformer architectures allocate independent, dense two-dimensional weight matrices to every feed-forward network (FFN) layer, coupling parameter count linearly with depth O(L). Conversely, weight-tied Looped Transformers decouple depth from parameters O(1) but enforce time-homogeneous updates across iterations. In this preprint, we formulate the L-layer FFN stack as a unified 3D Tensor Cuboid and introduce QuatTucker-ZigZagGPT, unifying three orthogonal mechanisms:1. 4D Quaternion Hamilton Algebra (H) for hypercomplex feature projections compiled into a single fused bfloat16 Tensor-Core GEMM (4x FFN compression).2. Block-Diagonal Tucker SwiGLU Cores enabling each depth slice to execute distinct 64x64 subspace rotations over a single shared high-rank feature basis.3. Braided ZigZag Diamond Routing (L0 -> L2 -> L1(h0 + h2) -> L3(h1 + h2)), introducing zero-FLOP, zero-parameter coarse-to-fine top-down refinement. We also introduce Annealed Gated Bounce-Back Scaffolding (GatedBounce), a sparsity-taxed training-time feedback path that self-distills to 0.0% inference overhead. Key Empirical Results:- FineWeb-Edu (GPT-2 BPE, NVIDIA L4, 3,000 steps): QuatTucker-ZigZagGPT-8 (26.5M total / 7.2M backbone params) achieves 5.1339 validation loss (169.68 PPL), outperforming a 33.5M (14.3M backbone) FlashAttention StandardGPT-8 baseline (5.2310 / 186.98 PPL) by -0.0971 loss (-17.30 PPL) with half the backbone parameters.- TinyStories (Tesla T4, 1.84M iso-params): CP-ZigZagLM achieves 2.48 PPL vs. 2.54 PPL for StandardTransformer-4.- Multi-Domain Compression: CP-CuboidNet achieves 92.2% on 8-bit Parity (3.2x smaller than ResNet4), 100.0% monotonic grokking on Mod-53 Addition (7.6x smaller than ResNet-8), 96.94% on Handwritten Digits (3.1x smaller), and 96.00% on 3-Turn Double Spirals (3.5x smaller).- Negative Results: We empirically establish that while FFNs thrive under Quaternion-Tucker sharing, attention projections must remain dense and independent per layer.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-21
DOI
https://doi.org/10.5281/zenodo.24424501
Primary Topic
Topic Modeling
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Quaternion-Tucker ZigZag Networks: Halving Transformer Backbone Parameters via Shared Hypercomplex Tensor Cores and Braided Top-Down Routing

Shubham Barmase
Zenodo (CERN European Organization for Nuclear Research)
Topic Modeling
preprint

Quaternion-Tucker ZigZag Networks: Halving Transformer Backbone Parameters via Shared Hypercomplex Tensor Cores and Braided Top-Down Routing

Shubham Barmase
preprint en

Abstract

Standard Transformer architectures allocate independent, dense two-dimensional weight matrices to every feed-forward network (FFN) layer, coupling parameter count linearly with depth O(L). Conversely, weight-tied Looped Transformers decouple depth from parameters O(1) but enforce time-homogeneous updates across iterations. In this preprint, we formulate the L-layer FFN stack as a unified 3D Tensor Cuboid and introduce QuatTucker-ZigZagGPT, unifying three orthogonal mechanisms:1. 4D Quaternion Hamilton Algebra (H) for hypercomplex feature projections compiled into a single fused bfloat16 Tensor-Core GEMM (4x FFN compression).2. Block-Diagonal Tucker SwiGLU Cores enabling each depth slice to execute distinct 64x64 subspace rotations over a single shared high-rank feature basis.3. Braided ZigZag Diamond Routing (L0 -> L2 -> L1(h0 + h2) -> L3(h1 + h2)), introducing zero-FLOP, zero-parameter coarse-to-fine top-down refinement. We also introduce Annealed Gated Bounce-Back Scaffolding (GatedBounce), a sparsity-taxed training-time feedback path that self-distills to 0.0% inference overhead. Key Empirical Results:- FineWeb-Edu (GPT-2 BPE, NVIDIA L4, 3,000 steps): QuatTucker-ZigZagGPT-8 (26.5M total / 7.2M backbone params) achieves 5.1339 validation loss (169.68 PPL), outperforming a 33.5M (14.3M backbone) FlashAttention StandardGPT-8 baseline (5.2310 / 186.98 PPL) by -0.0971 loss (-17.30 PPL) with half the backbone parameters.- TinyStories (Tesla T4, 1.84M iso-params): CP-ZigZagLM achieves 2.48 PPL vs. 2.54 PPL for StandardTransformer-4.- Multi-Domain Compression: CP-CuboidNet achieves 92.2% on 8-bit Parity (3.2x smaller than ResNet4), 100.0% monotonic grokking on Mod-53 Addition (7.6x smaller than ResNet-8), 96.94% on Handwritten Digits (3.1x smaller), and 96.00% on 3-Turn Double Spirals (3.5x smaller).- Negative Results: We empirically establish that while FFNs thrive under Quaternion-Tucker sharing, attention projections must remain dense and independent per layer.

Zenodo (CERN European Organization for Nuclear Research)
Topic Modeling
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon
Quaternion-Tucker ZigZag Networks: Halving Transformer Backbone Parameters via Shared Hypercomplex Tensor Cores and Braided Top-Down Routing — Shubham Barmase · Zenodo (CERN European Organization for Nuclear Research) (2026) | TGRS Research Map | TGRS