NeRF-Learner: A 2.79mJ/Frame NeRF-SLAM Processor with Unified Inference/Training Compute-in-Memory for Large-Scale Neural Rendering

This paper presents a NeRF-SLAM processor with online learning capability for real-time on-the-fly large-scale neural rendering. Cross-stack innovations are developed to push the efficiency, including 1) unified inference/training compute-in-memory for simultaneous NeRF inference and online training, 2) hierarchical and utilization-driven memory management for 3D map storage and access, and 3) temporal and spatial-aware ray-scheduler for efficient real-time rendering. The 22 nm NeRFLearner processor achieves 42.6FPS for large-scale 3D rendering, demonstrating a state-of-the-art $2.79 \mathrm{mJ} /$ frame and 36.3 TFLOPS/W macro efficiency, which is $6.2 \times$ higher than the SOTA NeRF accelerator. Furthermore, as the first CIM-based NeRF-SLAM processor, it achieves $4.2 \times$ system energy efficiency and $1.4 \times$ area efficiency improvement over the SOTA NeRF-SLAM processor.