Hidden Data Structures and Statistics of Natural Images

Complex, high-dimensional data are often analyzed by grouping their constituent units into components, sometimes referred to as latent features, which afford physical or biological interpretation. However, many different types of latent features and data decompositions can be defined, and one typically uses a trial-and-error approach to determine a decomposition that is natural to the system and its data. A principled understanding of which decomposition is appropriate for a given dataset is needed. In this work, we take a step in this direction and argue that sample-sample correlations carry important information to this effect. We construct a generative random latent feature matrix model of high-dimensional data based on linear mixing of latent features. As a key ingredient, we allow for a statistical dependence between the mixing coefficients. We argue that, with only two parameters (latent dimensionality and the structure of the statistical dependence), the resulting model captures characteristic properties found in many types of natural data. Specifically, the model can produce (overlapping) clusters, sparse mixing, and constrained (non-negative) mixing. We describe typical correlations and eigenvalue distributions of each pattern. Finally, we fit the model on correlation patterns in a benchmark natural images dataset, and we find a good match between the data and the sparse mixing regime of our model. This is in line with the well-known sparse coding structure in natural images, arguing that our approach identifies a correct latent decomposition of these data. We believe that our work will deliver similar insights for other diverse datasets.