Magnitude Attention-based Dynamic Pruning

Existing pruning methods often rely on weight importance to identify sparse structures but typically apply this information statically, without leveraging it adaptively during training. In this work, we propose a novel approach - M agnitude A ttention-based Dynamic P runing (MAP) method, which applies the importance of weights throughout both the forward and backward paths to explore sparse model structures dynamically. Magnitude attention is defined based on the magnitude of weights as continuous real-valued numbers enabling a seamless transition from a redundant to an effective sparse network by promoting efficient exploration. Additionally, the attention mechanism ensures more effective updates for important layers within the sparse network. Later, our approach shifts from exploration to exploitation, exclusively updating the sparse model composed of crucial weights based on the explored structure, resulting in pruned models that not only achieve performance comparable to dense models but also outperform previous pruning methods on CIFAR-10/100 and ImageNet. • We propose a novel magnitude attention-based pruning method (Section Section 4 ) and demonstrate its effectiveness by comparing it with competitive state-of-the-art pruning methods (Section Section 5.2 ). • We analyze the impact of our magnitude attention and provide insights on the importance of choosing the appropriate mask or weighting scheme in dynamic pruning (Section Section 5.3.1 ). • We introduce an exploration-exploitation strategy for dynamic pruning (Section Section 4 ). Notably, we observe that this approach yields highly effective results compared to other static pruning methods 1 1 We refer to the approach of updating only the important weights as static pruning, which contrasts with the concept of dynamic pruning (Lin et al., 2020 ; Guo et al., 2016). .