A Low-Latency Feed-Forward Architecture for Image Filtering via Row-by-Row Processing

This paper presents a novel accelerator architecture for real-time image filtering applications that require high throughput and low latency. The proposed architecture consists of two systolic arrays: a Row Convolution (RConv) Array and a Column Convolution (CConv) Array. The RConv Array processes one row of the input image and computes one row of an intermediate image per clock cycle which is then input to CConv Array. The CConv Array computes one row of the output image per clock cycle. A novel aspect of the architecture is that the line delays within the convolution architecture are used efficiently due to the parallelism. This leads to a significant reduction in the memory requirement and latency of the architecture. While the row convolution in the RConv Array is implemented using a direct-form FIR filter, the column convolution in the CConv Array is implemented using the transpose-form FIR filter. Thus, the processing elements in the CConv array are implicitly pipelined and do not require additional pipelining delays. The parallelized row-by-row processing significantly reduces the memory overhead per output pixel. The proposed architecture is then generalized where multiple rows of the image can be processed in a clock cycle, leading to further proportional reduction in latency and memory requirements. The proposed architecture is fully feed-forward and can be pipelined further as needed. We show that using a Xilinx Virtex-7 2000T FPGA clocked at 100 MHz, the architecture achieves <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$16.0\times$</tex-math> </inline-formula> , <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$86.3\times$</tex-math> </inline-formula> , and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$943\times$</tex-math> </inline-formula> improvements in the LUT-time <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$^2$</tex-math> </inline-formula> , FF-time <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$^2$</tex-math> </inline-formula> , and DSP-time <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$^2$</tex-math> </inline-formula> products over a baseline serial architecture for the non-separable case with Image size <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$128\times 128$</tex-math> </inline-formula> and filter size <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$11\times 11$</tex-math> </inline-formula> . The proposed architecture can operate at a throughput of 12.8 Gpixels/s whereas the baseline can only operate at 100 Mpixels/s.