From Rule Access to Rule Selection: Controlled Tests of Contextual Revision
High accuracy after anomalous evidence need not mean that a language model revised a rule: the same answer may already be its default. We separate rule access, default generalization, and evidence-sensitive selection in three synthetic microworlds with transformers of approximately 0.82M–4.79M parameters. A bidirectional test holds context inputs fixed while changing distance facts between two competing rules; answer permutations separately control the token multiset. Across 50 boundary-world checkpoints and ten context draws per query, group-median selection effects range from -0.002 to +0.013, despite high tag-given access in several arms. Given the two-rule hypothesis class, a symbolic selector identifies both rules perfectly. Three separately trained models of the same 0.82M architecture reach bidirectional accuracy 0.966–1.000 on 28 held-out direction queries after explicit episodic training. Corpus interventions explain a separate effect: full-range old-rule documents suppress alternative-rule defaults, whereas distance-only documents preserve direction accuracy. Completing a frozen low-dose experiment shows that 1–30 old-rule documents do not reproduce the collapse observed near 100 documents. These results localize the observed failure to evidence use under the tested training procedures, rather than information insufficiency or a demonstrated inability of the architecture. The contribution is a controlled diagnostic of rule selection and its training dependence, not a mechanism of scientific discovery.
Authors
- Dekun Yang (ORCID: https://orcid.org/0000-0001-8281-9674)
Institutions
- Zhejiang University (CN)
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-09-11
- DOI
- https://doi.org/10.5281/zenodo.22711856
- Primary Topic
- Topic Modeling
- Type
- preprint