TL-Fuser: Layout-Sensitive Scene Generation for Autonomous Driving

Recent advances in generating realistic images from image captions have enabled the synthesis of traffic scenes from textual descriptions. However, scene synthesis for autonomous driving is more challenging as the spatial relationship of objects in the traffic scene is important for decision making, and existing solutions fail in positional representation. To address this problem, in this paper we propose TL-Fuser, a novel framework for layout-sensitive scene generation under the guidance of textual description and layout constrains. In TL-Fuser, we propose a multimodal fusion approach that allows precise control over the position of objects in an image by integrating descriptive language with bounding box information. To enhance the consistency and fidelity of both semantics and layout in the generated images, a multi-scale attention mechanism is proposed for feature fusion. We conduct extensive experiments on the open-domain MSCOCO and COCO-Traffic datasets. The experimental results demonstrate that TL-Fuser outperforms GAN-based solutions for image synthesis, and shows superior performance in positional information representation against diffusion-based models and even visual large language models. This implies that TL-Fuser can be used to synthesize traffic scenes for the efficient testings of autonomous driving systems.