The high-dimensional structure of natural image representations varies systematically across visual cortex

The computational goal of the visual cortex is often described as systematic dimensionality reduction, where high-dimensional sensory input is gradually reduced to a low-dimensional manifold over multiple stages of processing. Recently, thanks to the unprecedented size of the Natural Scenes Dataset, we showed that the structure of human visual cortex representation is high-dimensional. We were able to reliably detect visual information encoded over many hundreds of latent dimensions. In an effort to reconcile these divergent theoretical predictions and empirical results, we set out to investigate how natural image representations are transformed along the visual hierarchy from a spectral perspective. Using a robust cross-decomposition approach, we estimated cross-validated covariance spectra of fMRI responses in several regions of interest in the visual cortex. In all of them, we observed power-law covariance spectra over hundreds of dimensions. Interestingly, we also noticed systematic trends: spectra decay more rapidly from earlier to later stages of visual processing. This could be seen from V1 to V4 and also from early- to mid- and late- stages of processing within the ventral, dorsal, and lateral visual streams. High-level functionally localized regions of visual cortex including face-, body-, scene- and object-selective cortex also show covariance spectra decaying more rapidly. Our findings demonstrate that while cortical representations of natural images are consistently high-dimensional across many stages of processing—thus using all available dimensions to encode visual information—there are, nonetheless, systematic regional variations in how information is concentrated along these dimensions. These differences in the representational structure of visual regions may provide insight into computational strategies in the human brain.