Img2Neuro: brain-trained neural activity encoders for enhanced object recognition

Abstract For decades, the brain’s visual pathway has inspired machine and deep learning models, yet these models oversimplify the brain’s complex visual processing. This is manifested in the significant superiority of the brain in comparison to the developed models in terms of the accuracy and the amount of data needed for training. Therefore, rather than using the brain as an inspiration, in this paper, we introduce Img2Neuro; a convolutional neural network model feature extractor that predicts the visual brain’s response to images by encoding neural activity. Img2Neuro is trained on natural scene images paired with single-neuron recordings from the visual cortex and thalamus of mice and monkeys. We explore the feasibility of using Img2Neuro as a feature extractor for object recognition, where the output of Img2Neuro in response to unseen images is used as input to classifiers with the task of recognizing the object in the image. We evaluated our approach on three benchmark datasets; namely, MNIST, Fashion-MNIST, and Cifar10. In our experiments, we examined the classification performance when Img2Neuro is used as a feature extractor compared to using the images as direct input to the classifier, using five different classifiers; namely, linear discriminant analysis, perceptron, logistic regression, ridge classifier, and a single-layer neural network. The results demonstrate superior performance when using Img2Neuro in most datasets and across all classifiers, reaching an enhancement in accuracy of 9% on the MNIST dataset, 2% on FashionMNIST, and 18% on Cifar10 in some cases compared to using raw images as an input in the classifiers. The performance enhancements suggest that brain-trained encoders can effectively capture image features for object recognition tasks. By leveraging neural response data, Img2Neuro demonstrates a promising avenue for bridging the gap between biological and artificial visual processing, ultimately leading to novel strategies for improving state-of-the-art object recognition models.