Codebase: Rethinking Business Process Simulation: A Utility-Based Evaluation Framework

Business process simulation (BPS) is a key tool for analyzingand optimizing organizational workflows, supporting decision-making byestimating the impact of process changes. The reliability of such esti-mates depends on the ability of a BPS model to accurately mimic theprocess under analysis, making rigorous accuracy evaluation essential.However, the state-of-the-art approach to evaluating BPS models hastwo key limitations. First, it treats simulation as a forecasting prob-lem, testing whether models can predict unseen future events. This failsto assess how well a model captures the as-is process, particularly whenprocess behavior changes from train to test period. Thus, it becomes diffi-cult to determine whether poor results stem from an inaccurate model orthe inherent complexity of the data, such as unpredictable drift. Second,the evaluation approach strongly relies on Earth Mover’s Distance-basedmetrics, which can obscure temporal patterns and thus yield misleadingconclusions about simulation quality. To address these issues, we pro-pose a novel framework that evaluates simulation quality based on itsability to generate representative process behavior. Instead of compar-ing simulated logs to future real-world executions, we evaluate whetherpredictive process monitoring models trained on simulated data performcomparably to those trained on real data for downstream analysis tasks.Empirical results show that our framework not only helps identify sourcesof discrepancies but also distinguishes between model accuracy and datacomplexity, offering a more meaningful way to assess BPS quality.