Logits Don't Lie: Pathway Specificity, Phonetic Scatter, and the Geometry of What Language Models Know
Language model output probability distributions directly reveal whether trained pathways exist in the model's weight matrices. At prompt tokens corresponding to known concepts, the model's predictions are near-deterministic; at fabricated concepts, predictions scatter across phonetic alternatives (d = 0.84, p < 0.001). This pathway specificity is measurable from outside the model using the Participation Ratio (PR = 1/Σp²) and Doodle Residual (DR), requiring only per-token probability distributions. We construct a three-stage inspection system — pre-generation recognition, generation monitoring with temporal signals, and aggregate scoring — validated across 18 models from 9 providers (4B–671B parameters) using 1,916 probes across 15 professional and regulated domains. The reading phase separates fabricated from genuine topics before any output is generated on 15 of 18 models (probe-level d = 0.16–1.30). Three temporal generation-phase signals achieve 18/18 coverage after architecture-specific corrections, including the three models where the reading signal is negligible or inverted. A frozen fusion of thirty-eight reading- and writing-phase geometry features, with length-correlated features excluded by design, attains leave-one-model-out AUC = 0.852 (median 0.853, minimum 0.704; 15 of 18 held-out models at or above 0.80) separating fabricated-topic from genuine-domain responses. Cross-domain validation demonstrates the measurement is domain-agnostic: legal document review (d = 3.43), financial fraud detection (d = 1.77), security monitoring on production telemetry (d = 2.054 on real Windows Event Log data; direction-agnostic validation across 14 models and 5 architectures, all d > 0.5), content authentication (d = 0.93), and manufacturing quality verification (d = 0.90) — all on a general-purpose model with zero domain-specific training. Additional applications include reasoning chain segmentation, through-thickness geometric profiling, training data composition detection, and a continuous memorization spectrum for IP forensics. The pathway specificity mechanism is argued to be structurally tamper-resistant: confident predictions at unfamiliar tokens require trained pathways, making score improvement indistinguishable from model improvement. Injection experiments support this argument, with architecture-dependent partial geometric shifts documented as a boundary condition. These findings support fourteen distinct applications and establish that external geometric measurement and internal mechanistic interpretability are complementary approaches to AI verification.
Authors
- Joseph Stephens
Institutions
- Spectral Labs (United States) (US)
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-09-11
- DOI
- https://doi.org/10.5281/zenodo.22699893
- Primary Topic
- Computational and Text Analysis Methods
- Type
- preprint