Learning and aligning single-neuron invariance manifolds in visual cortex

Understanding how sensory neurons exhibit selectivity to certain features and in-variance to others is central to uncovering the computational principles underlyingrobustness and generalization in visual perception. Most existing methods for char-acterizing selectivity and invariance identify single or finite discrete sets of stimuli.Since these are only isolated measurements from an underlying continuous mani-fold, characterizing invariance properties accurately and comparing them acrossneurons with varying receptive field size, position, and orientation, becomes chal-lenging. Consequently, a systematic analysis of invariance types at the populationlevel remains under-explored. Building on recent advances in learning continuousinvariance manifolds, we introduce a novel method to accurately identify and aligninvariance manifolds of visual sensory neurons, overcoming these challenges. Ourapproach first learns the continuous invariance manifold of stimuli that maximallyexcite a neuron modeled by a response-predicting deep neural network. It thenlearns an affine transformation on the pixel coordinates such that the same manifoldactivates another neuron as strongly as possible, effectively aligning their invari-ance manifolds spatially. This alignment provides a principled way to quantifyand compare neuronal invariances irrespective of receptive field differences. Usingsimulated neurons, we demonstrate that our method accurately learns and alignsknown invariance manifolds, robustly identifying functional clusters. When appliedto macaque V1 neurons, it reveals functional clusters of neurons, including simpleand complex cells. Overall, our method enables systematic, quantitative explo-ration of the neural invariance landscape, to gain new insights into the functionalproperties of visual sensory neurons.