Nuvpu: A $\mathbf{4. 8} \boldsymbol{\sim} \mathbf{9. 6} \sim \mathbf{m J} /$ Frame Progressive NTT-Based Unified Video Processor for Stable Video Streaming and Processing with Neural Video Codec

This paper presents NuVPU, a unified neural video processor that supports both streaming and post-processing with <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$4.8 \sim 9.6 \text{mJ} /$</tex> frame of energy efficiency. The Selective Convolution-mode Neural Engine (SCNE) adaptively selects either space or NTT convolution domain to increase throughput by <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$1.69 \sim 3.35 \times$</tex>. A Progressive NTT Unit (PNTU) lowers computation by 44.8 % and memory overheads by 80 % during domain changes. A Frequency-aware Compressor (FAC) and Adaptive Tile Scheduler (ATS) reduce the external memory access (EMA) of warping-based frame reuse by 81.3 %. NuVPU in 28nm CMOS process achieves \36.9 TOPS/W on UVG benchmark which is <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$9.2 \sim 2.3 \times$</tex> higher than previous video processors.