Neural Visual Synthesis for Graphics, 4D Gaussians and Videos

Efficient visual synthesis has played an indirect yet important role in the development of neural networks. Graphics processing units (GPUs) were initially designed for visual synthesis tasks such as rendering graphics. However, their parallel computing capabilities later made GPUs the ideal hardware for training neural networks. Since neural networks have greatly benefited from GPUs, a compelling question arises: How can neural networks, in turn, improve efficient visual synthesis? Existing visual synthesis methods still require improvement for higher efficiency. For example, Monte Carlo rendering takes over 15 minutes to generate a megapixel frame at a high-quality setting on a GPU, as computing thousands of samples per pixel is time-consuming yet essential for an offline path tracer. This thesis presents three technical contributions based on neural networks for efficient visual synthesis. We explore the performance improvement that neural networks can bring to different applications of visual synthesis, including graphics, 4D Gaussians, and videos. First, we propose speeding up Monte Carlo rendering by significantly reducing the number of pixels required for sampling. We introduce a neural network to predict future frames instead of rendering future frames. For each predicted future frame, certain pixels may not be accurately projected from existing frames due to challenging conditions, such as quick motion and large occlusion. Therefore, our method estimates a mask together with each subsequent frame. The estimated mask indicates the particular pixels requiring ray sampling to correct prediction results. Second, we introduce a feature splatting method based on 4D Gaussians, named Spacetime Gaussians. It is composed of three important components for dynamic view synthesis in a volumetric rendering way. The proposed Spacetime Gaussian is formulated by adapting 3D Gaussians with temporal opacity and parametric motion/rotation. This simple yet clean formulation enhances Spacetime Gaussian to capture static, dynamic, as well as transient content within a scene. Besides a new representation, we develop splatted feature rendering by replacing spherical harmonics with neural features. The neural features improve the rendering of appearance, which varies with viewpoint and time, while keeping the model compact. Additionally, with the guidance of training error and coarse depth, we sample new Gaussians in areas where current pipelines struggle to converge. Last, we apply neural visual synthesis to the general video prediction task. To produce high-quality results, we explore a trending architecture, Vision Transformers (ViTs). ViTs have made great progress in many low-level computer vision tasks like frame interpolation. However, video prediction is more challenging than video frame interpolation. Additionally, existing ViTs for video prediction are computationally expensive. Hence, we develop an efficient transformer for video prediction. We decouple temporal and channel features and employ transposed attention to reduce computation. We also implement a global query strategy to capture global information for motion. A depth shift module is introduced to better integrate cross-channel and temporal information, enabling efficient and high-quality video prediction.