A Real-time 4.31 mJ/Frame Neural-3DGS Processor with Voxel Similarity Memory Management and Opacity-based Sparsity Generation

This work presents an energy-efficient and real-time rendering Neural-3DGS processor for mobile AR/VR devices. While Neural-3DGS shows high-quality and fast rendering, it exhibits low energy efficiency & high latency in mobile implementation. The proposed processor has three key features for the overall processes in Neural-3DGS: 1) Voxel Similarity-aware Memory Management Unit (VSMMU) eliminates redundant operations and achieves 59.5%, 43.3% reduced energy for external memory access and neural-network computation. 2) LUT-based Pre-Sort Unit (LPSU) utilizes pre-computed order to reduce the latency of sorting by 64.3%. 3) Opacity-aware Gaussian Skipping Core (OGSC) exploits sparsity based on opacity and process 63.3% reduced MAC operations. The proposed processor is implemented in 28 nm CMOS technology. It achieves 98.8 FPS for real-time rendering and 4.31 mJ/Frame energy efficiency.