Towards an automated evaluation of design processes—an algorithm to predict critical situations during concept synthesis

Abstract In design research, there is a need for research methods that allow for larger numbers of participants in empirical studies, as small sample sizes lead to less statistically reliable results. The number of participants is limited by current research methods, such as protocol analysis, interviews, or analysis of eye tracking videos, because they require a lot of manual work during the evaluation. This is particularly noticeable in studies that analyze the cognitive processes of designers, e.g., during concept synthesis. Therefore, the goal of this paper is to develop an algorithm that automates the analysis of eye tracking data to predict designer perceived difficulties—critical situations in which methodical support could be beneficial—during design processes. By linking eye tracking data with retrospective think aloud data, a dataset was created for training different machine learning algorithms. The dataset was further processed, and three algorithms were evaluated regarding their suitability for automated detection of difficulties. The best algorithm, a cascading one-against-all classifier, achieved an accuracy of 62% and a false-negative rate of 26.6%. Depending on the type of difficulty, different eye tracking features were relevant for the decisions of the algorithms, highlighting the importance of tailored feature selection for each type of difficulty. The findings suggest that the automated analysis of eye tracking data using machine learning potentially facilitates larger studies and statistically more reliable findings, representing a significant step toward more efficient and insightful analysis of design cognition.