An 157TOPS/W Transformer Learning Processor Supporting Forward Pass Only with Zeroth-Order Optimization

This work presents the first Transformer learning processor that supports forward pass only with zeroth-order optimization. By applying uniform perturbation, sign-based weight gradient generation, and exponential ternary speculation, the training complexity is reduced by up to 96%. By employing parallel processing, tiled-based bitmask generation, and workload scheduling, the latency is reduced by up to 96%. Fabricated in 40-nm CMOS, the chip achieves a max energy efficiency of 157TOPS/W, outperforming prior backpropagation-based Transformer learning processors by 1.6-to-7.6×.