NeuGPU: An Energy-Efficient Neural Graphics Processing Unit for Instant Modeling and Real-Time Rendering on Mobile Devices

Neural radiance field (NeRF) is an emerging computer graphics task that is used for 3-D modeling and rendering in the metaverse, providing a user-friendly and immersive experience. It successfully replaces conventional 3-D modeling methods such as photogrammetry thanks to its simplicity, photo-realistic rendering quality, and versatility. However, it has limitations to be accelerated on mobile augmented reality (AR)/virtual reality (VR) devices due to its memory-intensive hash encoding and extensive computational load. Previous NeRF application specified integrated circuits (ASICs) only supported high frame-rate rendering while being incapable of immediate 3-D model creation. In this article, we present neural graphics processing unit (NeuGPU) to achieve both instant 3-D modeling and real-time rendering using NeRF with three key features. First, segmented hashing with spatial pruning (SHSP) partitions hash table (HT) into multiple segments and reduces 66% of external memory access (EMA). Spatial management unit (SMU) supports SHSP operation through three types of sub-block management. Second, attention-based hybrid interpolation unit (AHIU) optimizes heterogeneous memory access characteristics of hash-based NeRF, achieving 56.4% power reduction. Third, similarity-sparsity skipping (S3) core supports energy-efficient multi-layer perceptron (MLP) operation by applying coarse-grained skip for both similar and sparse data. As a result, NeuGPU is fabricated in a 28-nm process and occupies a 20.25-mm2 die area. Evaluated on the synthetic NeRF dataset, NeuGPU finally achieves <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$8.7{\times }$ </tex-math></inline-formula> faster modeling and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$231.4{\times }$ </tex-math></inline-formula> smaller energy per iteration compared with edge GPU even though it has <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$31{\times }$ </tex-math></inline-formula> smaller external memory bandwidth.