Logits Don't Lie: Pathway Specificity, Phonetic Scatter, and the Geometry of What Language Models Know

Language model output probability distributions directly reveal whether trained pathways exist in the model's weight matrices. At prompt tokens corresponding to known concepts, the model's predictions are near-deterministic; at fabricated concepts, predictions scatter across phonetic alternatives (d = 0.84, p < 0.001). This pathway specificity is measurable from outside the model using the Participation Ratio (PR = 1/Σp²) and Doodle Residual (DR), requiring only per-token probability distributions. We construct a three-stage inspection system — pre-generation recognition, generation monitoring with temporal signals, and aggregate scoring — validated across 18 models from 9 providers (4B–671B parameters) using 1,916 probes across 15 professional and regulated domains. The reading phase separates fabricated from genuine topics before any output is generated on 15 of 18 models (probe-level d = 0.16–1.30). Three temporal generation-phase signals achieve 18/18 coverage after architecture-specific corrections, including the three models where the reading signal is negligible or inverted. A frozen fusion of thirty-eight reading- and writing-phase geometry features, with length-correlated features excluded by design, attains leave-one-model-out AUC = 0.852 (median 0.853, minimum 0.704; 15 of 18 held-out models at or above 0.80) separating fabricated-topic from genuine-domain responses. Cross-domain validation demonstrates the measurement is domain-agnostic: legal document review (d = 3.43), financial fraud detection (d = 1.77), security monitoring on production telemetry (d = 2.054 on real Windows Event Log data; direction-agnostic validation across 14 models and 5 architectures, all d > 0.5), content authentication (d = 0.93), and manufacturing quality verification (d = 0.90) — all on a general-purpose model with zero domain-specific training. Additional applications include reasoning chain segmentation, through-thickness geometric profiling, training data composition detection, and a continuous memorization spectrum for IP forensics. The pathway specificity mechanism is argued to be structurally tamper-resistant: confident predictions at unfamiliar tokens require trained pathways, making score improvement indistinguishable from model improvement. Injection experiments support this argument, with architecture-dependent partial geometric shifts documented as a boundary condition. These findings support fourteen distinct applications and establish that external geometric measurement and internal mechanistic interpretability are complementary approaches to AI verification.

Authors

Institutions

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-09-11
DOI
https://doi.org/10.5281/zenodo.22699893
Primary Topic
Computational and Text Analysis Methods
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
preprint

Logits Don't Lie: Pathway Specificity, Phonetic Scatter, and the Geometry of What Language Models Know

Joseph Stephens
Zenodo (CERN European Organization for Nuclear Research)
Computational and Text Analysis Methods
preprint

Logits Don't Lie: Pathway Specificity, Phonetic Scatter, and the Geometry of What Language Models Know

Joseph Stephens
preprint en

Abstract

Language model output probability distributions directly reveal whether trained pathways exist in the model's weight matrices. At prompt tokens corresponding to known concepts, the model's predictions are near-deterministic; at fabricated concepts, predictions scatter across phonetic alternatives (d = 0.84, p < 0.001). This pathway specificity is measurable from outside the model using the Participation Ratio (PR = 1/Σp²) and Doodle Residual (DR), requiring only per-token probability distributions. We construct a three-stage inspection system — pre-generation recognition, generation monitoring with temporal signals, and aggregate scoring — validated across 18 models from 9 providers (4B–671B parameters) using 1,916 probes across 15 professional and regulated domains. The reading phase separates fabricated from genuine topics before any output is generated on 15 of 18 models (probe-level d = 0.16–1.30). Three temporal generation-phase signals achieve 18/18 coverage after architecture-specific corrections, including the three models where the reading signal is negligible or inverted. A frozen fusion of thirty-eight reading- and writing-phase geometry features, with length-correlated features excluded by design, attains leave-one-model-out AUC = 0.852 (median 0.853, minimum 0.704; 15 of 18 held-out models at or above 0.80) separating fabricated-topic from genuine-domain responses. Cross-domain validation demonstrates the measurement is domain-agnostic: legal document review (d = 3.43), financial fraud detection (d = 1.77), security monitoring on production telemetry (d = 2.054 on real Windows Event Log data; direction-agnostic validation across 14 models and 5 architectures, all d > 0.5), content authentication (d = 0.93), and manufacturing quality verification (d = 0.90) — all on a general-purpose model with zero domain-specific training. Additional applications include reasoning chain segmentation, through-thickness geometric profiling, training data composition detection, and a continuous memorization spectrum for IP forensics. The pathway specificity mechanism is argued to be structurally tamper-resistant: confident predictions at unfamiliar tokens require trained pathways, making score improvement indistinguishable from model improvement. Injection experiments support this argument, with architecture-dependent partial geometric shifts documented as a boundary condition. These findings support fourteen distinct applications and establish that external geometric measurement and internal mechanistic interpretability are complementary approaches to AI verification.

Zenodo (CERN European Organization for Nuclear Research)
Spectral Labs (United States) (US)
Quality Education, Peace, Justice and strong institutions
Computational and Text Analysis Methods
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.