TerraFusion: Joint generation of terrain geometry and texture using latent diffusion models

Three-dimensional terrain models are essential in domains such as video game development and film production. Because surface color is often correlated with terrain geometry, capturing this relationship is critical for generating realistic results. However, most existing methods synthesize either a heightmap or a texture without adequately modeling their inherent correlation. We propose a method that jointly generates terrain heightmaps and textures using a latent diffusion model. First, we train the model in an unsupervised manner to randomly generate paired heightmaps and textures. Then, we perform supervised learning on an external adapter to enable user control via hand-drawn sketches. Experiments demonstrate that our approach supports intuitive terrain generation while preserving the correlation between heightmaps and textures. Our method outperforms the two-stage and GAN-based baselines by ensuring structural coherence, in which textures naturally align with geometry, successfully accommodating both realistic landscapes and extreme user-defined shapes.