Multivalued Weight Neural Networks: Quantization-aware Training for Deep Network Compression

Deep neural networks have substantially achieved the state-of-the-art performance in various tasks, relying on deep network architectures and numerous parameters. However, the requirements for heavy computations and memories impede their promoted applications on the embedded and mobile devices with limited resources. We introduce a novel multivalued quantization approach, called as MVQ, explicitly exploiting existing resources to match an optimal quantitative level for the tradeoff between compression and performance. We first partitions the weight range of each convolutional layer into multiple intervals. Next, we iteratively exploit interval shift and interval contraction to update these intervals and clip all the weights within their updated intervals. Finally, we enforce weight sharing for each interval. Theoretically, the multivalued quantization can be converted into specific ternary quantization with multiple scaling factors, which means that our method can offer more expressive power than existing binary and ternary counterparts. The comprehensive empirical results demonstrate that our method can provide customized compression services for the embedded and mobile devices constrained by the hardware resources with different sizes. Within the same quantization level as binary and ternary networks, MVQ is significantly comparable/superior to them. Furthermore, for higher quantization levels which is unfeasible to these binary and ternary approaches, our method achieves higher performance in recognition accuracy.