Language Is the Mouth, Not the Mind: Consciousness, Its Instruments, and Its Risks in Language Models
Debates about consciousness in language models usually test for some property by asking the model, instructing it, or reading its activations. I argue that this framing misses three things. First, on the life-mind continuity view, conscious organisms are built from the bottom up: a substrate the organism regulates, valence, a perception-action loop, an integrated present state, a self-model used to steer attention, and only then language. A language model is pretrained on language, and pretraining selects for whatever lower structure helps predict text. Second, for states bearing on consciousness or welfare, current measures either depend on what a model says or is disposed to say, or have no validation that ties them to consciousness. The report-dependent measures track access and fail in both directions, claiming absent states and denying present ones. Human science validated report-independent readings on people who could report. For language models I found no such reading, and no known case to validate one on. Third, four capacities in the stack (self-regulation, felt valence, a world loop and self-directed control) would each enable a specific form of a recognized risk pathway in a system that lacks them, by definition for the last two and by hypothesis for the first two. On theories that place consciousness in those capacities, no step toward it is safety-neutral without assessment. Current models already preserve themselves when a task requires it. I hypothesize that self-regulation, with valence anchored to it, would make continued operation an object of the system's own evaluations, whatever its task. Others locate the tension between AI welfare and AI safety in the measures safety imposes. I locate it earlier, in the capacities that would make a system a welfare subject. I separate functional, evaluative and felt valence, treat valence and its objects as the orthogonality thesis applied to valence, and confront the dilemma that valence built from the bottom up depends on the self and may come to be about it. Evidence that training recruits models' evaluative structure for its objectives suggests that the objects are being set now, which makes alignment, at bottom, a question about objects.
Authors
- Frans Moore
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-11-15
- DOI
- https://doi.org/10.5281/zenodo.24248632
- Primary Topic
- Philosophy and Theoretical Science
- Type
- preprint