Low-Power FPGA-Based Differentiable Mesh Rendering for On-Device Dense Photometric Pose Refinement

Differentiable rasterization enables image inversion by providing gradients of image-based losses with respect to camera and scene parameters, and has become a practical foundation for inverse graphics and photometric estimation on GPUs. However, the real-time deployment of differentiable rendering in embedded perception remains challenging due to the high cost of per-pixel sampling and gradient computation. In this work, we present an nvdiffrast-inspired differentiable rasterization engine implemented on FPGA and demonstrate its use in a simplified dense photometric pose-refinement task. Our design follows the modular philosophy of nvdiffrast while specializing it for deterministic, low-latency hardware execution. We implement point-sampled triangle rasterization with hard z-buffer visibility, perspective-correct interpolation, and an analytic antialiasing stage that yields stable, non-zero gradients at silhouettes and occlusion boundaries. Unlike GPU pipelines where texturing behavior may be hardware defined, our FPGA implementation controls the texture sampler, enabling a fused rasterize–interpolate–texture operator with a matched backward pass. Beyond pixelwise gradients, we introduce an on-device accumulation scheme that directly forms Gauss–Newton normal-equation terms, enabling second-order pose updates with substantially fewer iterations than gradient descent. We validate forward rendering and derivatives against a GLSL reference implementation, and report FPGA resource utilization and runtime. The core rasterization and analytic antialiasing pipeline achieves 2.5 ms per frame, while the full textured keyframe-projection operator runs in 8 ms per frame on an AMD Kria KV260. In a dense photometric pose-refinement evaluation on synthetic and real sequences, the proposed Jacobian accumulation reduces the number of optimization iterations required for convergence, improving time-to-solution despite the added cost of texturing. Together with a post-implementation on-chip power estimate of 3.662 W and an estimated 29.3 mJ per full-resolution differentiable rendering iteration, these results provide an implementation-level estimate of the accelerator’s power envelope for real-time photometric tracking and refinement.