Governed Semantic Tokens: Tokenization as an Admissibility Boundary

Large Language Models increasingly operate in contexts where outputs do more than inform: they trigger actions, authorize decisions, and persist as binding state. Yet the boundary at which raw signals—text, pixels, events—are allowed to count remains largely ungoverned. Tokenization is treated as plumbing: a compression step optimized for throughput, not legitimacy. This paper introduces Governed Semantic Tokens (GST), a framework that reframes tokenization as an admissibility boundary—the jurisdiction where meaning is either lawfully admitted or refused. Rather than opaque, model-specific vocabularies, GST employs registry-backed token identifiers whose semantics are explicit, typed, versioned, and governed. Tokens become units of admissible meaning, not merely units of prediction. By relocating governance to the tokenizer layer, GST enables structural resistance to prompt injection, cross-model semantic stability, auditable promotion to action, and refusal as a first-class success condition. The framework is model-agnostic and applies uniformly across modalities, treating text, vision, UI events, and tool interactions as inputs to a single governed semantic substrate. The paper positions GST as MCIR-0: the minimal viable substrate beneath higher-order governed systems, where meaning itself must be admitted before reasoning or action can proceed. Tokenization, long assumed neutral, is shown to be constitutional infrastructure.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-02-03
DOI
https://doi.org/10.5281/zenodo.18469506
Primary Topic
Ethics and Social Impacts of AI
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
article

Governed Semantic Tokens: Tokenization as an Admissibility Boundary

Adam Ableman Mazurk
Zenodo (CERN European Organization for Nuclear Research)
Ethics and Social Impacts of AI
article

Governed Semantic Tokens: Tokenization as an Admissibility Boundary

Adam Ableman Mazurk
article en

Abstract

Large Language Models increasingly operate in contexts where outputs do more than inform: they trigger actions, authorize decisions, and persist as binding state. Yet the boundary at which raw signals—text, pixels, events—are allowed to count remains largely ungoverned. Tokenization is treated as plumbing: a compression step optimized for throughput, not legitimacy. This paper introduces Governed Semantic Tokens (GST), a framework that reframes tokenization as an admissibility boundary—the jurisdiction where meaning is either lawfully admitted or refused. Rather than opaque, model-specific vocabularies, GST employs registry-backed token identifiers whose semantics are explicit, typed, versioned, and governed. Tokens become units of admissible meaning, not merely units of prediction. By relocating governance to the tokenizer layer, GST enables structural resistance to prompt injection, cross-model semantic stability, auditable promotion to action, and refusal as a first-class success condition. The framework is model-agnostic and applies uniformly across modalities, treating text, vision, UI events, and tool interactions as inputs to a single governed semantic substrate. The paper positions GST as MCIR-0: the minimal viable substrate beneath higher-order governed systems, where meaning itself must be admitted before reasoning or action can proceed. Tokenization, long assumed neutral, is shown to be constitutional infrastructure.

Zenodo (CERN European Organization for Nuclear Research)
Openalex Percentile: Top 70%
Ethics and Social Impacts of AI
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Governed Semantic Tokens: Tokenization as an Admissibility Boundary — Adam Ableman Mazurk · Zenodo (CERN European Organization for Nuclear Research) (2026) | TGRS Research Map | TGRS