ULM-Bench: A Preregistered Benchmark and Audit Protocol for Ontological Coherence in Language Models
This record contains the complete artifact bundle for the arXiv preprint ULM-Bench: A Preregistered Benchmark and Audit Protocol for Ontological Coherence in Language Models (Mark JJ McCormack, independent researcher).Contents: - ULM v1.0 — a versioned Hegelian category graph (Logic → Nature → Spirit) with 654-slot grid, 555 uncontested verified slots, 53 pending, 28 gaps, 18 contested, 342 application mappings, and 24 cross-cutting Universals. Released as JSON (ulm_v1.json), CSV position tables, raw exports, and JSON Schema. - OMS Reference Implementation — deterministic Python reference for the Ontological Match Score (oms_reference.py), computing direction, distance, and triad-integrity losses against the sequestered ULM graph. - Semantic Logic Mapping (SLM) Specification — formal parser spec (PARSER_SPEC.md), relation inventory, worked examples, failure taxonomy, and JSON Schema (slm.schema.json). - ULM-Bench Benchmark Specification — three task families (Static Retrieval, Cross-Domain Normative Dilemmas, Ontological Rupture), item schema, four conditions (Ungrounded, BFO/CCO, GraphRAG, ULM), six controls, and eight primary endpoints. - Preregistered Confirmatory Protocol — \$25k study design with hypotheses (H-01–H-19), power analysis (N=194/condition for 10pp effect), go/no-go gates (Parser F1 ≥ 0.85, κ ≥ 0.67, OMS AUC ≥ 0.80), and falsification criteria. - Pilot Results — 200 items × 4 conditions; L condition 52% OMS pass (0.707 mean), U/B/G 100% abstain. - Audit Scripts — ulm_audit.py (14,940 checks, 0 errors), build_ulm.py (deterministic builder). - Gold Sets & Annotation Manual — 50-sentence gold set with adjudicated SLMs (κ = 0.82), coding manual.License: Sublation License v4.0 (this record). The preprint is licensed separately under arXiv's terms.Status: Proposal and preregistration. No confirmatory experiments have been run. All artifacts are released for independent verification and execution. ------------------------------------------------------------ Status: Proposal and preregistration. Simulations have been run. No confirmatory experiments have been run. All artifacts released for independent verification and execution under the Sublation License v4.0. Citation: McCormack, M.J.J. (2026). ULM-Bench: A Preregistered Benchmark and Audit Protocol for Ontological Coherence in Language Models. Zenodo: https://doi.org/10.5281/zenodo.23324479 Keywords: language model evaluation, ontological coherence, benchmark, preregistration, Hegel, dialectical logic, AI safety, deception detection, semantic parsing, Anthropic, Safety By Design, Safety-By-Design, BFO, Basic Formal Ontology, gBFO, Genuine Basic Formal Ontology, AI Alignment, ASI Alignment, Super Intelligence, Artificial Super Intelligence, AI Priors, Aristotle, Plato,
Authors
- McCormack Mark
Institutions
- University of Alberta (CA)
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-11-20
- DOI
- https://doi.org/10.5281/zenodo.24405603
- Primary Topic
- Topic Modeling
- Type
- preprint