Controllable 3D Urban Generation via Conditional VAEs and CLIP for Simulations

The 3D urban generation technology is of great significance in the game industry, autonomous driving simulation, and traffic simulation. However, existing image-based methods generate results with blurred boundaries, graphic-based methods produce results with poor diversity, and rendering-based methods synthesize inconsistent images. To solve these problems, this paper proposes a novel controllable 3D urban generation pipeline, which consists of urban layouts and 3D scenes generation. Specifically, we introduce the graph dynamic attention model and the dummy node to enhance the diversity of the generated results. We also design a scene asset matching method based on the CLIP model to controllably generate 3D urban scenes. Experimental results indicate that the diversity of the generated results surpasses that of existing methods, and the generated 3D urban scenes are of good consistency.