Resource-efficient DL Model Inference with Weight Clustering and Zero-skipping

Implementation of deep learning models is constrained by different memory and computational requirements. While significant progress has been made in enhancing the processing element performance, the complete system throughput largely depends on memory transfer from the off-chip interfaces. This work proposes a complete on-chip solution for accelerator implementation on devices with resource constraints. We propose batch-wise clustering of the weights of DL models and in-situ generation of weights to be fed to the acceleration for the computations as per the model specifications. This architectural change is further complemented by the popular technique of zero-skipping. Experimental observations indicate that an average reduction in model parameters of more than 26% can be obtained at a 10% reduction in accuracy for three benchmark models extensively used in the literature.