Characterizing Design Rationales Using Computational Linguistics and Human Evaluations

Abstract Design rationales capture the explicit justifications behind design decisions. Often, rationales vary in the content and depth of information, making the study and comparison of rationales challenging. This project aims to characterize design rationales and develop a standardized approach to assess design rationale quality at scale. In total, 2250 rationales were machine-generated across two different representations and evaluated by two raters across five dimensions of quality. Rationales were then characterized using natural language processing techniques, resulting in 108 linguistic features for each rationale. The linguistic features were used to build predictive models for each quality dimension. The main results identify correlations between linguistic features and human evaluations, show that structured rationales were rated higher than unstructured rationales across the five dimensions, and present a model to predict rationale quality for new texts. These findings help inform strategies to improve human- and machine-generated rationales. The predictive tool offers a scalable method for evaluating different representations of rationale, thereby supporting more effective documentation practices.