A 16nm 231mW Channel MLP Optimized Neural Processing Unit for 80.3% Top-1 ImageNet Classification Accuracy at 102fps

Channel multi-layer-perceptron (MLP) is playing a crucial role in today’s neural networks, such as Transformer. Furthermore, MetaFormer [5] demonstrates that with separated channel MLPs, simply using pooling layers as spatial token mixers (i.e. PoolFormer, PF) exhibit exceptional performance as well, as shown in Fig. 1. Despite the advantage in terms of complexity and accuracy, we identify three design challenges for efficient deployment on edge devices. Firstly, massive external memory access (EMA) is induced due to the channel expansion in MLP blocks. Secondly, processing throughput could be degraded by the computation latency of feature statistics for layer norm (LN) and the low MAC utilization for a largekernel CONV $7 \times 7$ layer. Thirdly, classification accuracy could be degraded by fixed-point implementation for LN and Gaussian Error Linear Unit (GELU). In this work, we present a high-performance processor to address the challenges with three key features:1) EMA reduction with bottleneck-based layer fusion (BBLF) and model-divided batch mode (MDBM) for MLP; 2) throughput improvement with LN pipelined scheduling (LNPS) and kernel reshaping (KRS) for CONV77;× 3) complexity-efficient and accuracy-preserved fixed-point LN and difference-LUT-based GELU.