Invariant Ordinal Cores: How Transformers Represent, Compare, and Route Continuous Physical Scales
Algorithmic Core Extraction (ACE) recently demonstrated that transformers solve discrete tasks—Markov state transitions, modular addition, subject–verb agreement, and indirect object identification—through compact, causally necessary and sufficient subspaces shared across seeds and architectures. Yet much of real-world reasoning operates not on discrete symbols, but on continuous physical and semantic orderings: ranking planets by diameter, animals by mass, chemical elements by atomic number, or countries by land area. Here we extend ACE to continuous ordinal ranking across pretrained language models (Qwen2.5-1.5B, Qwen3-1.7B, and Qwen3-0.6B), introducing paired-swap ACE to cancel prompt-template variance and strictly causal pre-context entity extraction to isolate intrinsic entity representations before a comparison target enters the causal attention window. Across models and domains, ordinal comparison decomposes into a shared three-stage algorithmic core pipeline: (1) an intrinsic midlayer entity magnitude register (layers 9–12, 32–43% depth) where a single 1D direction linearly encodes physical log-scale—recovering planetary log10 (diameter) (𝑟 = 0.923–0.994) and atomic number log10 (𝑍) across 20 chemical elements (𝑟 = 0.942–0.966, unseen elements 𝑟 = 0.977)—and permits single-scalar counterfactual transplants; (2) a late-layer lexicon-invariant quantitative comparator bottleneck (layers 19–21, 68–75% depth), where a 1D core extracted from only six planets transfers zeroshot across unseen planets, animals (heavier/lighter), and numbers (higher/lower, Flip AUC = 0.021, 𝜏 = −1.00); and (3) lexical readout divergence (layers 22–28), where vocabulary-specific heads fan out while propositional Yes/No remains orthogonal. Finally, we turn this mechanistic pipeline into CoreRank, a zero-fine-tuning 𝑂(𝑁) pointwise ranking readout from the Stage-1 entity core. Across 3 models and 3 strictly held-out real-world domains under clean and adversarial distractor prompts, CoreRank raises unseen-entity accuracy from 52.7% to 72.0% overall (57.6% to 81.8% on quantitative scales; AUC 0.753 → 0.903), unlocks Qwen3-0.6B from chance (50.0%, AUC 0.351) to 80.0% (0.895 AUC) on unseen elements, cuts prompt-distractor verdict flips 5.7× (25.5% → 4.5%), and mathematically eliminates position bias (67.3% → 0.0%) and intransitivity cycles (54.4% → 0.0%).
Authors
- Marcio Diaz
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-11-21
- DOI
- https://doi.org/10.5281/zenodo.24413357
- Primary Topic
- Topic Modeling
- Type
- article
- Field-Weighted Citation Impact
- 0.00