Data-driven performance measures using global properties of attractors for testing black-box surrogate models of chaotic systems

In climate systems, physiological models, optics, and many more, surrogate models are developed to reconstruct chaotic dynamical systems. We introduce four data-driven measures for low-dimensional systems using global attractor properties to evaluate the quality of the reconstruction of a given time series from a surrogate model. The measures are robust against the initial position of the chaotic system as they are based on empirical approximations of the correlation integral and the probability density function, both of which are global properties of the attractor. In contrast to previous methods, we do not need a manual fitting procedure, making the measures straightforward to evaluate. Compared with the n-dimensional Wasserstein distance, the measures are fast to evaluate, while compared with the Hausdorff distance, they perform better. Furthermore, we introduce a statistical framework inspired by hypothesis testing to systematically find and reject surrogate models whose reconstructions significantly differ from the true system. Furthermore, we show that the measures can be used as a statistical ranking metric, which, in practice, allows for hyperparameter optimization. Applying our measures to reservoir computing with a low number of nodes, we demonstrate how the measures can be used to reject poor reconstructions and use them as a tool to find an optimal spectral radius for which considerably fewer solutions are rejected and the overall quality of the solution improves.