Optoelectronic Pipeline Architecture of Convolutional RNN for Energy Efficient Inference at the Speed of Light

This paper proposes an optoelectronic architecture of Convolutional Recurrent Neural Network (C-RNN in short). It employs RNN layers that replace area-consuming fully connected (FC) layers in typical convolutional neural network (CNN) architectures. The convolution and RNN layers in this architecture process input data in a pipelined manner that improves the throughput of the inference processing. It takes advantage of both the high input compression capabilities of CNNs and the compact and power-efficient nature of RNNs. The proposed optoelectronic C-RNN architecture achieves over 97.8% accuracy on the MNIST dataset while maintaining the advantages of power-efficient and high-speed characteristics of photonics. Our proposed optoelectronic C-RNN architecture can reach 240 TOPs/W, which is ten times more efficient than CMOS-based dedicated CNN accelerators.