Implicit Generative Modeling by Kernel Similarity Matching

Understanding how the brain encodes stimuli has been a fundamental problem in computational neuroscience. Insights into this problem have led to the design and development of artificial neural networks that learn representations by incorporating brain-like learning abilities. Recently, learning representations by capturing similarity among input samples has been studied (Pehlevan et al., 2018) to tackle this problem. This approach, however, has thus far been used only to learn downstream features from an input and has not been studied in the context of a generative paradigm, where one can map the representations back to the input space, incorporating not only bottom-up interactions (stimuli→latent) but also learning features in a top-down manner (latent→stimuli). We investigate a kernel similarity matching framework for generative modeling. Starting with a modified sparse coding objective for learning representations proposed in prior work (Olshausen & Field, 1996; Tolooshams & Ba, 2021), we demonstrate that representation learning in this context is equivalent to maximizing similarity between the input kernel and a latent kernel. We show that an implicit generative model arises from learning the kernel structure in the latent space and show how the framework can be adapted to learn manifold structures, potentially providing insights as to how task representations can be encoded in the brain. To solve the objective, we propose a novel alternate direction method of multipliers (ADMM)-based algorithm and discuss the interpretation of the optimization process. Finally, we discuss how this representation learning problem can lead toward a biologically plausible architecture to learn the model parameters that ties together representation learning using similarity matching (a bottom-up approach) with predictive coding (a top-down approach).