EnvMap-GS: two-stage outdoor Gaussian reconstruction with background-to-environment map baking

Abstract Reconstructing outdoor environments from “inside-out” captures, where a camera moves within a restricted area but looks outward, remains challenging due to the presence of both well-textured nearby regions and low-detail distant backgrounds. We introduce a two-stage Gaussian Splatting framework that explicitly separates and optimizes these regions, yielding higher-fidelity novel view synthesis and allowing the replacement of the distant part with a high-quality, inpainted environment map to speedup the rendering process. In stage one, background primitives are initialized within a spherical shell and optimized using a loss that combines a background-only photometric term with two geometric regularizers: one constraining Gaussians to remain inside the shell, and another one aligning them with local tangential planes. In stage two, foreground Gaussians are initialized from a Structure-from-Motion reconstruction, added and refined using the standard rendering loss, while the background set remains fixed but contributes to the final image formation. Background Gaussians can be rendered to an object-free environment map that is inpainted to fill missing parts and can replace the Gaussian-based background for faster rendering. Experiments on diverse outdoor datasets show that our method reduces background artifacts and improves perceptual quality of novel view renderings compared to state-of-the-art baselines, including the removal of floaters in the navigation region.