Constructing Complexity Metrics for Measuring Generalization Ability of Deep Learning Models

Deep neural networks have achieved tremendous success in many applications, but their generalization ability remains unexplained. Understanding this can enhance their use in safety-critical applications and guide better model design. A key aspect of this is studying model complexity, which is related to generalization. This paper proposes a technique for constructing complexity metrics to measure the generalization of deep learning models, focusing on convolutional neural networks for image classification tasks. Firstly, we consider complexity metrics based on model structure from statistical learning theory and use them as baseline metrics. Secondly, we summarize and generalize previous metrics based on model parameter norms and outputs. Thirdly, we propose a complexity metric based on the sharpness of model solutions, combining PAC-Bayesian theory to measure the expected sharpness of model local minima. Finally, experiments on two model sets are conducted to verify the effectiveness of each metric by examining the relationship between complexity and model generalization, thereby constructing the final set of model complexity metrics. We then analyze the correlation between these candidate metrics and the model's generalization gap, selecting highly correlated metrics to construct a complexity indicator set. Experimental results demonstrate that the proposed metrics based on sharpness and PAC-Bayes theory achieve the best correlation results on two model sets, PGDL-CIFAR10 and PGDL-SVHN.