Analysing the bias introduced by adaptive designs to estimates of psychometric functions

An adaptive design adjusts dynamically as information is accrued. In psychometrics and psychophysics, a class of studies investigates a subject's ability to perform tasks as a function of the stimulus intensity, ie the amount or clarity of information supplied for the task. The relationship between performance and intensity is represented by a psychometric function. Such experiments routinely apply adaptive designs using both previous intensities and performance to assign stimulus intensities, the strategy being to sample intensities where information about the psychometric function is maximised. We investigate the influence of adaptation on statistical inference about the psychometric function focusing on estimation, considering parametric and non-parametric estimation under both fixed and adaptive designs and under within-subject independence as well as dependence. We study the scenarios analytically and numerically through a simulation study. We show that while asymptotic properties of estimators are preserved under adaptation, the adaptive nature of the design introduces small-sample bias, in particular in the slope parameter of the psychometric function. We supply an explanation of this phenomenon that formalises and supplements the one found in the literature. We argue that this poses a dilemma for studies applying an adaptive design in the form of a trade-off between more efficient sampling and the need to increase the number of samples to ameliorate small-sample bias.