Ultralow Power In-Sensor\nNeuronal Computing with Oscillatory\nRetinal Neurons for Frequency-Multiplexed, Parallel Machine Vision

In-sensor and near-sensor computing architectures enable\nmultiply\naccumulate operations to be carried out directly at the point of sensing.\nIn-sensor architectures offer dramatic power and speed improvements\nover traditional von Neumann architectures by eliminating multiple\nanalog-to-digital conversions, data storage, and data movement operations.\nCurrent in-sensor processing approaches rely on tunable sensors or\nadditional weighting elements to perform linear functions such as\nmultiply accumulate operations as the sensor acquires data. This work\nimplements in-sensor computing with an oscillatory retinal neuron\ndevice that converts incident optical signals into voltage oscillations.\nA computing scheme is introduced based on the frequency shift of coupled\noscillators that enables parallel, frequency multiplexed, nonlinear\noperations on the inputs. An experimentally implemented 3 × 3\nfocal plane array of coupled neurons shows that functions approximating\nedge detection, thresholding, and segmentation occur <i>in parallel</i>. An example of inference on handwritten digits from the MNIST database\nis also experimentally demonstrated with a 3 × 3 array of coupled\nneurons feeding into a single hidden layer neural network, approximating\na liquid-state machine. Finally, the equivalent energy consumption\nto carry out image processing operations, including peripherals such\nas the Fourier transform circuits, is projected to be <20 fJ/OP,\npossibly reaching as low as 15 aJ/OP.