Can LLMs Turn Design Discussions into Architectural Tests? An Empirical Study with Codestral

This is a preprint version of the contribution submitted to ECSA 2026. It has not undergone peer review or post-submission revisions.Architectural drift remains a persistent challenge as system implementations often diverge from their intended designs. While detection tools exist, they require formalizing constraints in specific languages—a process that is often costly and time-consuming. Meanwhile, much architectural knowledge emerges during development through informal discussions, such as pull request comments on platforms like GitHub, where design rules are expressed in natural language but remain disconnected from automated verification.In this paper, we explore using Large Language Models (LLMs) to bridge this gap by transforming informal artifacts into executable representations. We conducted an empirical study using the Codestral model to convert 10,500 design rules extracted from GitHub into Java tests via the ArchUnit library. We evaluated the generated checks based on syntactic validity, API usage, semantic alignment, and violation detection.Our results show that the model consistently produces syntactically valid code, with 67% of the outputs showing partial or full alignment with the original rule. Additionally, 58% of the checks correctly applied the ArchUnit API. These findings highlight the significant potential of LLMs to turn informal architectural knowledge into executable verification, while also identifying key reliability challenges that remain for future improvement.