Developing Design Theory Using Large Language Model Agents: A Case Study of C-K Theory

Abstract Computational simulations have long been used for design and engineering analysis, but there is also a rich history of using simulations to develop design theory. The present work draws inspiration from design, management, psychology, and other fields that have used simulations to develop theory and integrates recent advances in large language models (LLMs). While design theory has been used to improve the development of generative design tools, this study goes the other direction and uses generative AI-based simulations to further develop design theory. We take C-K theory as a case study to demonstrate this approach. The simulations used computational agents fueled by large language models designed to adhere to the C-K theory methodology in both wording and framework. These were evaluated using a mixed-methods approach across three studies, including two simulation experiments and one conventional content analysis of qualitative, LLM-generated, C-K transition rationale statements. The results reveal that concept-to-concept transitions were the predominant C-K operation and that concept diversity trended downward across various experimental conditions. These results are in contrast to the generic generativity described in human-based C-K theory and highlight gaps between C-K as a design theory and C-K in a generative AI-based computational simulation form, suggesting future directions for operational and theoretical development. Ultimately, this case study demonstrates that design theories can be simulated computationally using LLM agents, albeit with differences and potential limits when compared to humans, thus providing the design research community with a new approach to further developing design theories.