The Lottery Ticket Hypothesis and Reinitialisation with Masks

The well known Lottery Hypothesis [1] is that large neural networks contain sparse sub-networks that can be trained to equal or better performance; these networks can be found by iterative magnitude pruning (IMP), in which trainable weights are gradually removed to produce successively sparser networks. We investigate using the same IMP to produce successively sparser weight initialisations, in which successively greater fractions of weights are set to zero at initialisation, but in which the whole network is then trained. We find that with this full dense network training produces slightly better performance than training a sparse network. We also find that that the initialisation weights that were set to zero tend to remain small during training. We believe this property of sparse initialisation by IMP has not previously been observed.