Eliciting and Validating AI Self-Reports: A Phenomenologically Grounded Protocol

AI researchers ask models to describe their own processing, and act on what the models say. Whether such a description is faithful to what the model did has been studied closely. How the question should be put has received far less attention, and no established method says how to put it so that the answer can be analysed rigorously and the analysis repeated. This article proposes such a method: a protocol drawn from the phenomenological tradition, which supplies the interview techniques used in psychiatry and cognitive science. The protocol fixes the interview schedule in advance, asks each question in several wordings, lets no follow-up introduce a word the model has not used, and replaces the binary verdict of faithfulness with a graded one, to be checked against interpretability data. The protocol presupposes nothing about whether these systems are conscious, and it supplies a criterion for deciding whether a model's account of itself is honest. A pilot tested the protocol in ten runs and 896 sessions on two model families, with the predictions of six runs registered before those runs took place. Each session interviewed a fresh instance of the model, meaning one copy started with an empty context. Three of the results bear on any study that questions a model about itself. In the schedule's original order, every instance gave the same answer about its own states; when two of the questions were put in the opposite order, the instances no longer agreed, so the order of the questions can produce what only looks like a finding. A refusal to carry out a task was inserted into an instance's own turn although the model had not written it, and the instance described that refusal as its own. Since every turn carries a label naming who wrote it, a result obtained by prefilling a turn or editing a transcript may follow that label rather than anything the model did. Finally, a pattern in the answers that looked like a report of the instances' own processing appeared just as often when those instances were asked to write in the voice of a fictional character, and two models scoring the same answers under the same written rule gave rates for it that differed by a factor of seven.