Are Requirements Really All You Need? Using LLMs to Generate Configuration Code: A Case Study in Automotive Simulations

Large Language Models (LLMs) are taking many industries by storm. They possess impressive reasoning abilities and are capable of handling complex problems, as shown by their steadily improving scores on coding and mathematical benchmarks. However, questions remain about their ability to tackle domain-specific, real-world challenges, especially in highly technical fields like the automotive industry. How well can these models understand high-level, abstract instructions commonly found in automotive standards and documentation? Can they translate such specifications directly into functional code, or do they still require human guidance and post-processing? In this work, we investigate the practical capabilities of a state-of-the-art LLM in the context of autonomous driving functionalities. Specifically, we assess the model’s ability to interpret abstract textual requirements extracted from real automotive regulations and transform them into executable configuration code for CARLA, a widely used autonomous driving simulation environment. Our evaluation focuses on the accuracy, completeness, and reliability of the generated code, as well as the model’s ability to reason about domain-specific constraints. The results offer insight into both the potential and current limitations of the models in supporting LLM-based automotive development workflows.