Reversed depth representation in human and artificial visual systems

Stereopsis facilitates the brains of animals with front-facing eyes in linking the left and right retinal light patterns to disentangle the complex depth information in the sensory input. Extracting depth structures has been thought to be sufficiently explained by the binocular energy model in the primary visual cortex. However, engaging in real-world 3D tasks requires more complex stereo computations in higher visual areas. The locations and mechanisms through which the brain transforms the physical stimuli representation (binocular disparity) into perceptual representation (depth perception) remain unclear. To address the issue, we combined human psychophysics, neuroimaging, and deep neural network (DNN) simulations. We designed random-dot stereograms (RDSs) by varying the binocular dot contrast correlation to reverse the physical and the perceived depth: RDSs whose physical disparity elicited near were perceived as far, and vice versa. Participants (N=22) reported perceiving depth in reverse when presented with the engineered RDSs in a two-alternative forced choice (2AFC) task discriminating near and far. Decoding analysis on their fMRI voxel response patterns (V1-3, V3A, V3B, hMT, hV4) revealed that only V3A represented reversed depth, suggesting that reversed depth is an extrastriate phenomenon involving more complex stereo computation beyond early areas. We tested two DNNs to gain insight into the network architecture underlying reversed depth in V3A. One network could learn the geometry of contextual background for regressing disparity from a rectified pair of stereo images. The second network did not include such contextual information in its disparity estimation. Both networks were trained on the SceneFlow datasets, which provide accurate depth maps. We demonstrate that the network incorporating contextual information exhibited similar behavioral performance in the human’s depth judgment tasks. We conclude that V3A may house the neuronal circuit that learns the spatial context to generate neuronal activity that gives rise to conscious reversed depth perception.