Low Rank Factorizations are Indirect Encodings for Evolved Neural Networks

Deep neuroevolution is a highly scalable alternative to reinforcement learning due to its unique ability to encode network updates in a small number of bytes. Recent insights from traditional deep learning indicate high-dimensional models possess intrinsic, low-rank structure. In this work, we introduce low-rank, factorized neuroevolution-an indirect encoding through which we can search a small space of low-rank factors that enforce underlying structure across a network's weights. We compare our approach with non-factorized networks of similar and smaller size to understand how much performance can be attributed to the smaller search space. We evaluate our method on a language modeling task using transformers. Our study shows that low-rank, factorized neuroevolution outperforms or is competitive with non-factorized neuroevolution, performing notably well on language modeling. More broadly, these results show how we can use insights from backpropgation-based methods to enhance neuroevolution.