Model Reduction Using a Hybrid Approach of Genetic Algorithm and Rule-based Method

Model reduction is a technique that reduces the computational resources required to run a model (neural network) by pruning parameters or structures in the model. Most of the model reduction algorithms achieve the goals of model reduction and accuracy preserving through multiple iterations of pruning-retraining process. However, this retraining process is quite time-consuming, making the model size reduction algorithm particularly inefficient, especially when the parameters of model exceed tens of millions. In this paper, we propose a hybrid approach combining genetic algorithm (GA) and rule-based method. With the integration of GA and a rule-based method, the time cost of searching for a well-performing model can be significantly reduced. This strategy greatly reduces GA’s search space and time cost. With a very limited number of retraining epochs (<10), the accuracy and pruning ratio (sparsity) of the reduced model can catch up the results of state-of-the-art. We conduct experiments on a gesture recognition model with over 30 million parameters. The experimental results show that for this model, our approach prunes 74.6% of the parameters with 3.8% accuracy drop without retraining. With only three epochs of retraining, our approach prune 93.1% of the parameters without any accuracy drop.