Tensor networks and efficient descriptions of classical data

We investigate the potential of tensor-network-based machine-learning methods to scale to large image and text datasets. We study how the mutual information between a subregion and its complement scales with the subsystem size <a:math xmlns:a="http://www.w3.org/1998/Math/MathML"><a:mi>L</a:mi></a:math>, similarly to approaches used in quantum many-body physics. Using various mutual information (MI) estimation methods, including an autoregressive network-based estimator, we find that simple image datasets exhibit area law scaling, suggesting efficient representation by two-dimensional tensor networks. More complex image datasets exceed the area law, indicating the need for generalized tensor network states or hybrid models. For Wikipedia text data, we observe power-law scaling <b:math xmlns:b="http://www.w3.org/1998/Math/MathML"><b:mrow><b:mi>I</b:mi><b:mrow><b:mo>(</b:mo><b:mi>L</b:mi><b:mo>)</b:mo></b:mrow><b:mo>∝</b:mo><b:msup><b:mi>L</b:mi><b:mi>ν</b:mi></b:msup></b:mrow></b:math> with <c:math xmlns:c="http://www.w3.org/1998/Math/MathML"><c:mrow><c:mi>ν</c:mi><c:mo>≈</c:mo><c:mn>0.82</c:mn></c:mrow></c:math>, approaching a volume law. This implies that one-dimensional tensor networks with area law entanglement may not efficiently capture the structure of text. We introduce two models to reproduce this scaling: a quantum-inspired random pair toy model and a linguistically motivated Markovian dependency tree model. In the latter model, matching the observed MI scaling allows us to infer the word-word correlation length distribution in text.