From a Measure of Confidence to a Measure of the Level of Knowledge

Confidence degrees assigned by respondents to their responses are generally taken at their face value. An experiment where respondents were asked to indicate twice their confidence in their (changed or unchanged) response has, however, showed that those confidences can greatly vary over time at the individual level. I propose a model that takes that variation into account and considers confidence as a latent variable - the level of knowledge - to be estimated through a true score approach. The model is defined in the special case of a scale with a given number of confidence degrees. It assumes that when faced with this type of testing requirements, a person experiences uncertainty in a way that can be represented by a finite set of partial knowledge states. It leans mainly on a conditional independence assumption. As the model is intractable under that sole assumption, additional testable and simple constraints must be imposed on the way confidence errors are distributed. The model was applied to data collected in the experiment. The results show that, under a general (population) overestimation bias, very different individual profiles are hidden with different distributions of errors. The model enables also to make predictions about one single individual by only examining his (her) calibration errors. Some errors patterns observed on the replicated data can indeed be anticipated with the proposed models.