On learning what to learn: Heterogeneous observations of dynamics and establishing possibly causal relations among them

Before we attempt to (approximately) learn a function between two sets of observables of a physical process, we must first decide what the <i>inputs</i> and <i>outputs</i> of the desired function are going to be. Here we demonstrate two distinct, data-driven ways of first deciding "the right quantities" to relate through such a function, and then proceeding to learn it. This is accomplished by first processing simultaneous heterogeneous data streams (ensembles of time series) from observations of a physical system: records of multiple <i>observation processes</i> of the system. We determine (i) what subsets of observables are <i>common</i> between the observation processes (and therefore observable from each other, relatable through a function); and (ii) what information is <i>unrelated</i> to these common observables, therefore particular to each observation process, and not contributing to the desired function. Any data-driven technique can subsequently be used to learn the input-output relation-from k-nearest neighbors and Geometric Harmonics to Gaussian Processes and Neural Networks. Two particular "twists" of the approach are discussed. The first has to do with the <i>identifiability</i> of particular quantities of interest from the measurements. We now construct mappings from <i>a single</i> set of observations from one process to <i>entire level sets</i> of measurements of the second process, consistent with this single set. The second attempts to relate our framework to a form of causality: if one of the observation processes measures "now," while the second observation process measures "in the future," the function to be learned among what is common across observation processes constitutes a dynamical model for the system evolution.