Exploring the Fractal Space of Sparse Deep Neural Network

Sparse deep neural networks (DNNs) are preferable in many real applications, especially in edge computing. Howto find the sparse structure of deep neural networks so that the model can achieve better performance is an important research topic. Under the same sparsity level, the connection patterns of DNNs affect their performance significantly. In this paper, we study the relationship between the structure and the performance of DNNs from a graph perspective. Our research focus on fractal structure, since it has already been demonstrated that many real-world networks benefit from their high fractal dimension structures. We statistically analyze the correlation between fractal dimension and the performance of sparse DNNs, whose structures are constructed by relational graphs. The graphs we investigate are generated from fractal growth model, Barabási–Albert (BA) model, Erdős–Rényi (ER) model,Watts-Strogatz (WS) model, regular network model, stochastic block model and random geometric network model. We find that fractal dimension of the preset structure is positively correlated with the performance of corresponding DNNs, indicating that fractal dimension could be a key factor in designing sparse DNNs of high effectiveness.