Precision, Fitness, Generalization, and Simplicity as Quality Dimensions for Decision Discovery Algorithms

Operational decisions significantly influence organizational performance and individual well-being. Decision mining offers a method to discover and analyze decision logic from decision logs, enhancing decision-making processes. However, evaluating the quality of decision discovery algorithms remains a challenge. While precision, fitness, generalization, and simplicity are well-established quality dimensions in process mining, their adaptation to the decision mining domain is underexplored. This study adapts these four dimensions to the necessary characteristics of decision models, providing a framework for evaluating decision discovery algorithms. Using a design science research approach, we develop tailored metrics and functions and demonstrate their application through a practical example of environmental permit management modeled in Decision Model and Notation (DMN). Precision measures how the discovered decision model reproduces the observed fact types and values from the decision log, detecting over-specification in the decision model. Fitness evaluates how completely the decision model covers the behavior in the decision log, identifying missing or under-specified elements in the decision model. Generalization assesses the model’s robustness to unseen decision cases by quantifying how well the discovered decision model performs beyond the training data. Simplicity captures the complexity in the discovered decision model in relation to a human actor-specified threshold. These insights guide decision model improvements, contributing to higher transparency, accountability, and fairness in operational decision-making processes. This research bridges a gap in the body of knowledge by providing a concrete methodology for evaluating decision discovery algorithms. The results support organizations in aligning decision models with regulatory requirements and public values, while also laying a foundation for future research.