Modeling optic flow tuning in MSTd with convolutional neural networks

Performance-optimized convolutional neural networks (CNNs) have emerged as highly effective models at predicting neural responses in brain areas along the primate ventral stream, but it is largely unknown whether they effectively model neurons in the complementary primate dorsal stream. We explored how well CNNs model the optic flow tuning properties of neurons in dorsal area MSTd and we compared our results with the Non-Negative Matrix Factorization (NNMF) model proposed by Beyeler, Dutt, & Krichmar (2016), which successfully models many tuning properties of MSTd neurons. To better understand the role of computational properties in the NNMF model that give rise to MSTd-like optic flow tuning, we created additional CNN model variants that implement key NNMF constraints — non-negative weights and sparse coding of optic flow. While the CNNs and NNMF models both accurately estimate the observer’s self-motion from purely translational or rotational optic flow, NNMF and the CNNs with nonnegative weights yield substantially less accurate estimates than the other CNNs when tested on more complex optic flow that combines observer translation and rotation. Despite their poor accuracy, however, neurons in the networks with the nonnegativity constraint give rise to tuning properties that align more closely with those observed in primate MSTd. Interestingly, the addition of the sparsity constraint has a negligible effect on the accuracy of self-motion estimates and model tuning properties. Across all models, we consistently observe the 90-degree offset in the preferred translation and rotation directions found in MSTd neurons, which suggests that this property could emerge through a range of potential computational mechanisms. This work offers a step towards a deeper understanding of the computational properties and constraints that describe optic flow tuning primate area MSTd.