The Spectral Ideal Observer: Focus Error and Pupil Size Estimation

Perception science has long appreciated that amplitude and phase relationships carry critically important information about behaviorally-relevant properties of proximal stimuli and the distal environment, ranging from edge structure, to focus error, to fixation error (binocular disparity), to face identity. But it has not been clear how to make optimal use of phase information. Here, we report a new class of image-computable ideal observers—spectral ideal observers—that make optimal use of both amplitude and phase. They operate on the real and imaginary coefficients of a stimulus’ Fourier transform—rather than on amplitude and phase, or on pixels—, by characterizing the joint probability distributions of these coefficients across frequency and the latent variable of interest. They show, quantitatively, the information phase provides over-and-above amplitude alone. Significant computational advantages are obtained. For noise images (e.g. 1/f noise), the coefficients are conditionally Gaussian; for natural images, appropriate normalization Gaussianizes them. From these distributions, the posterior over the latent variable, or the optimal Bayesian-theoretic decision variable, can be computed. The results provide principled predictions of human performance, and of the supporting neural computations. Spectral ideal observers are well-suited for problems in which the transformation from scene- to image-space is naturally modeled as a (shift-variant) convolutional operation. To demonstrate the effectiveness of spectral ideal observers, we develop one for the joint task of estimating focus error and pupil size from individual images blurred by the optics of human eyes. We show that focus error and pupil size can be accurately estimated with exquisite precision from L- and S-cone responses to individual images; performance differences can be predicted on an eye-by-eye basis. In addition to this application in optics, the spectral ideal observer should have broad application to other estimation and discrimination tasks, including binocular disparity in stereo-depth perception and binaural sound localization in audition.