Geometric sparsification in recurrent neural networks

Sparse neural networks are neural models with pruned weight matrices, which can be both more efficient and more interpretable than dense models. The structures that underlie effective sparse architectures, however, are poorly understood. In this paper, we propose a new technique for sparsification of recurrent neural networks (RNNs), called moduli regularization. Moduli regularization imposes a geometric relationship between neurons in the hidden state of the RNN parameterized by a manifold. We further provide an explicit end-to-end moduli learning mechanism, in which optimal geometry is inferred during training. We verify the effectiveness of our scheme in three settings, testing in navigation, natural language processing, and synthetic long-term recall tasks. While past work has found some evidence of local topology positively affecting network quality, we show that the quality of trained sparse models also heavily depends on the global topological characteristics of the network.