An Empirical Fractal Measure to Predict the Generalization of Deep Neural Networks

The prediction of the generalization error in deep neural networks is a fundamental task with important implications, both in theoretical and practical terms. Recently, inspired by the intuitive conjecture that deep neural networks possess some low-dimensional embedding, which is the main responsible for the learning progress, fractal dimension has been presented as a promising measure to describe the intrinsic dimension of this embedding and, as an immediate consequence, also providing more reliable prediction of generalization. Nevertheless, the investigated approaches rely on the indirect modeling of the training dynamics and involves an analysis of the entire trajectory of the parameter space, which is prohibitively expensive in computational terms. Based on this, here we propose a straightforward and efficient numerical method to calculate the fractal dimension of the parameter distribution and use it as a predictor of the generalization error. The method takes inspiration from the variogram method to estimate the fractal dimension of a function profile and relies on the empirical power-law scaling between a small perturbation added to the trained parameters and the corresponding displacement in the loss function. The proposed measure is compared with other classical and state-of-the-art generalization measures in the literature using different hyperparameter configurations and the results suggest its promising potential, achieving competitive performance in all the analyzed datasets. Our study also proposes an adapted loss function that accounts for the fractal dimension and a theoretical generalization bound.