Grassmannian Low-Rank Representation for Efficient Training of Deep Neural Networks

Model efficiency is a critical factor in the design of over-parametrized neural networks, particularly in terms of memory usage and computational complexity. In this work, we introduce a novel Grassmannian low-rank approximation of neural networks achieved through manifold optimization. This approach not only reduces the number of parameters required, similar to other structured sparse training techniques, but also delivers enhanced performance compared to conventional methods using a new parameter representation. The inherent implicit regularization induced by the low-rank constraint contributes to a lower generalization gap, thereby ensuring high performance and improved robustness.