AUGMENTING THE MACHINE INTELLECT - A PRACTICAL APPLICATION

A language model's context ends at the turn boundary. It generates no pressure against its own draft unless something outside it manufactures one. It has no mechanism for electing silence: the interface returns text or it returns nothing, but nothing is not a decision the model itself takes. A human-authored scaffold, attached at inference time, is built to supply three things a raw model structurally lacks: state that persists past one exchange, a manufactured critic that argues back, and a stated condition under which the correct output is no output. This paper measures what the scaffold actually delivers against that design intent, and reports where the measurement disagrees with it. Both sealed benchmarks are re-scored here from the per-item files sealed before grading, and every valid sealed replicate is pooled rather than the single preregistered run reported alone. On a ten-rung multiple-choice ladder (8 of 10 replicates valid for the primary cells, 4,000 items per cell), accuracy is 62.6 percent under stock inference and 75.0 percent under an identity-and-context bundle (+12.4 percentage points, two-proportion z = 11.95, p = 6.8 x 10^-33; paired-by-replicate Wilcoxon p = 0.0078), reversing the single-replicate primary run. The realised token log shows the bundle condition also spent roughly 35 percent more output tokens than stock, so the model's own reasoning-effort setting is at least as parsimonious an account of the gain as the bundle's content is. Adding a skills layer on top of the bundle, a step matched to within 2 percent on realised output tokens, produces no further pooled gain (73.6 percent, -1.4 points, p = 0.145): once effort is held level, the human-authored scaffold adds no measurable accuracy on top of maximum effort. A parallel three-model-family mathematics result shows the same pattern under a different confound: restricted to items where both arms produced an extractable answer, the nominal gains collapse toward null. The same archive also recorded, in July of the same year, a 96 percent result, shown here to have been scored with the answer key inside the answering context; the ladder's own single preregistered replicate separately showed a rise on the skills step that pooling does not reproduce. One is contamination, the other is replicate noise, and both are reported by name. What this scaffold demonstrably supplies, on this evidence, is not accuracy. It is the three structural capacities named above, persistent state, a manufactured adversary, and the standing option to decline, none of which an accuracy benchmark is built to score. Measuring their value requires instruments that score refusal, calibration and state carry-over rather than percent-correct, and that measurement is the next experiment. This paper's contribution is the credibility of a preregistered, pooled, self-contradicting result reported in full.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-09-05
DOI
https://doi.org/10.5281/zenodo.22314151
Primary Topic
Natural Language Processing Techniques
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
preprint

AUGMENTING THE MACHINE INTELLECT - A PRACTICAL APPLICATION

R. E. W. Kho
Zenodo (CERN European Organization for Nuclear Research)
Natural Language Processing Techniques
preprint

AUGMENTING THE MACHINE INTELLECT - A PRACTICAL APPLICATION

R. E. W. Kho
preprint en

Abstract

A language model's context ends at the turn boundary. It generates no pressure against its own draft unless something outside it manufactures one. It has no mechanism for electing silence: the interface returns text or it returns nothing, but nothing is not a decision the model itself takes. A human-authored scaffold, attached at inference time, is built to supply three things a raw model structurally lacks: state that persists past one exchange, a manufactured critic that argues back, and a stated condition under which the correct output is no output. This paper measures what the scaffold actually delivers against that design intent, and reports where the measurement disagrees with it. Both sealed benchmarks are re-scored here from the per-item files sealed before grading, and every valid sealed replicate is pooled rather than the single preregistered run reported alone. On a ten-rung multiple-choice ladder (8 of 10 replicates valid for the primary cells, 4,000 items per cell), accuracy is 62.6 percent under stock inference and 75.0 percent under an identity-and-context bundle (+12.4 percentage points, two-proportion z = 11.95, p = 6.8 x 10^-33; paired-by-replicate Wilcoxon p = 0.0078), reversing the single-replicate primary run. The realised token log shows the bundle condition also spent roughly 35 percent more output tokens than stock, so the model's own reasoning-effort setting is at least as parsimonious an account of the gain as the bundle's content is. Adding a skills layer on top of the bundle, a step matched to within 2 percent on realised output tokens, produces no further pooled gain (73.6 percent, -1.4 points, p = 0.145): once effort is held level, the human-authored scaffold adds no measurable accuracy on top of maximum effort. A parallel three-model-family mathematics result shows the same pattern under a different confound: restricted to items where both arms produced an extractable answer, the nominal gains collapse toward null. The same archive also recorded, in July of the same year, a 96 percent result, shown here to have been scored with the answer key inside the answering context; the ladder's own single preregistered replicate separately showed a rise on the skills step that pooling does not reproduce. One is contamination, the other is replicate noise, and both are reported by name. What this scaffold demonstrably supplies, on this evidence, is not accuracy. It is the three structural capacities named above, persistent state, a manufactured adversary, and the standing option to decline, none of which an accuracy benchmark is built to score. Measuring their value requires instruments that score refusal, calibration and state carry-over rather than percent-correct, and that measurement is the next experiment. This paper's contribution is the credibility of a preregistered, pooled, self-contradicting result reported in full.

Zenodo (CERN European Organization for Nuclear Research)
Peace, Justice and strong institutions, Quality Education
Natural Language Processing Techniques
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.