Improved Neural Network Compression using Generative Adversarial Networks

Deep neural networks are typically computationally expensive, thus, there is a strong motivation to develop less intensive models while maintaining performance. Distilling knowledge from a teacher network to a lightweight student network is a critical application of neural network compression where the student network can learn meaningful feature space and perform comparably to the teacher network, thereby, eliminating wasteful redundancy in the teacher network. In this study, we propose a mean squared error loss metric in addition to the binary cross entropy in the learning stage. We further demonstrate the impact of using embeddings from an earlier stage in the Teacher Network to train the Generative Adversarial Network (GAN) model. Results over standardized MNIST dataset show an increase in accuracy by 3% when using an earlier dense embedding layer for training the GAN model.