Evaluating Design Rationale

Abstract Design rationale captures the justification behind a design decision. Often, rationales vary in the content and depth of information, making the study and comparison of rationales challenging. This project aims to characterize design rationale and develop a computational approach to evaluate design rationale quality at scale. In total, 2250 rationales were machine-generated using GPT across two different representations, and a portion of the rationales (n = 512) were evaluated by two raters across five dimensions of quality. Rationales were then characterized using natural language processing techniques, resulting in 108 linguistic features for each rationale. The evaluations and linguistic features were used to build eight predictive models for each quality dimension. The main results show that structured rationales were rated higher than unstructured rationales across the five dimensions. Thus, the tested feature, specification, and evidence (FSE) framework was shown to be a worthwhile approach to represent the justification behind a design decision. Moreover, key linguistic features that correlate with higher quality ratings were identified. Future work will explore how design rationale quality and characteristics impact design decisions, particularly in a human-AI teaming context where generative design recommendations could benefit from including generative design rationales.