Comparing apples to oranges from photoreceptors to inferior temporal cortex through neural manifolds

In the neural processing of color, cone photoreceptor outputs are combined over successive stages to form inferior temporal cortex (IT) cells with narrower color tuning and more diverse peak preferences. To understand the benefits of these transformations, we explored the capability of photoreceptor, retinal ganglion cell and IT population responses to distinguish between pairs of natural fruits and vegetables, based solely on their color distributions extracted from hyperspectral images. Simulations with realistic Poisson response variability show that linear separations are possible for many more pairs by IT neurons. Shapes of low-dimensional manifolds extracted from neural population responses explain the improvement.