PaletteGaussian: 3D Photorealistic Color Editing with Gaussian Splatting

3D editing, particularly involving realistic color editing, plays a crucial role in various multimedia domains, such as augmented reality and filmmaking. Traditional 3D reconstruction methods encounter challenges in achieving high-fidelity reconstruction for complex scenes. In recent years, methods based on implicit 3D representations, like Neural Radiance Fields (NeRF), have demonstrated effectiveness in rendering complex scenes. However, these methods face difficulties in interactively editing scene colors and often exhibit slow processing speeds. Addressing these challenges, we propose the PaletteGaussian framework for interactive color editing and real-time rendering based on a palette and 3D Gaussian Splatting (3DGS). First, we introduce a two-stage training strategy to ensure rendering quality and enhance the accuracy of object extraction in the scene. Next, we present an image-driven learning-based approach, I-learning, for convenient interactive color editing driven by both images and text. Finally, we perform parameter baking to achieve real-time rendering. In summary, PaletteGaussian supports two editing levels, scene-level and object-level, offering three interaction modes: manual, image-driven, and text-driven editing. It enables high-resolution real-time rendering. Our comprehensive experiments demonstrate that PaletteGaussian exhibits efficient performance, diverse interaction modes, and realistic color editing.