Natural retinal cone distributions emerge from optical and neural limits to vision

Biological visual systems develop highly efficient solutions in response to physical limitations. In particular, the human retinal cone mosaic supports both high spatial and color vision acuity in an imperfect optical environment. Here, we show that naturalistic cone distributions can emerge from simple constraints such as chromatic aberration (CA) and naturalistic behavioral task performance. We model key components of the visual system with a CA-constrained optical simulation, learnable cone mosaic sampling and a state-of-the-art deep artificial neural network. We also designed a custom dataset, ImageNet-Bird, by selecting images that require both high spatial acuity and color acuity for correct classification. By training our model to perform this visual task, we show that the model’s emerged cone mosaic resembles a cone mosaic found in humans. One important characteristic is the relative deficiency of S cones compared to M and L cones. Moreover, in a performance comparison experiment using fixed cone mosaics with different S cone ratios, we showed that the performance is consistently better when the S cone ratio is lower. Finally, we also observed that artificial neural networks have a different set of limitations from biological neural systems due to inductive biases imposed by the network architecture; for example, the model requires stationarity in the cone mosaic to achieve a good performance. More generally, our results serve as a concrete instance in which the functional organization of vision is driven by inherent optical and neural limitations, and may provide a new framework for understanding observed statistics of the visual system.