On the Eigenstructure of the Fisher Information Matrix and Its Role in Generalization Error

To characterize generalization performance by analyzing the training dynamics of natrual gradient descent, we present theoretical analyses of a modified natural gradient descent method, leveraging the neural tangent kernel theory. Firstly, we provide an analytical expression for the function acquired through this modified natural gradient descent under the assumptions of an infinite network width limit and a Gaussian conditional output distribution. Subsequently, by interpreting the generalization error as stemming from the distribution discrepancy between the training data and the true data, we propose a criterion for modification in the eigenspaces of the Fisher information matrix to reduce the generalization error bound. Through this approach, we establish that modifying the training direction of the neural network by modifying the eigenstructure of Fisher information matrix in function space leads to a reduction in generalization error. Moreover, in a special and important case of distribution discrepancy due to noise, our therotical framework gives an explanation to some existing method that muted the small eigenvalues during training. The theoretical results are also illustrated through numerical experiments.