Evaluation of GenAI as an accelerator in process mining

Recent progress in artificial intelligence has enabled the development of new Process Mining (PM) techniques. While new methods have been investigated, it is still unclear which process mining activities can be feasibly augmented with Generative Artificial Intelligence (GAI), nor have the implications for individuals’ confidence in and responsibility for the results been clarified. A mixed-methods approach was applied, including a systematic literature review to identify process mining tasks suitable for enhancement. Experts assessed these tasks based on relevance and accessibility for novices. These tasks were then used in a user experiment to evaluate the performance, confidence, and responsibility measures of Large Language Model (LLM) augmentation compared to conventional tools. The experiment concluded with interviews to gather qualitative insights. Although statistical analysis did not reveal any significant effects on responsibility and accuracy, a possible trend towards higher confidence when working with LLMs compared to traditional tools was identified. A significant statistical effect for task completion speed was found, demonstrating that LLMs can, in fact, improve efficiency. Furthermore, some participants seemingly blindly trusted the LLM’s output without a thorough examination of the results. It was found that LLM-augmented tasks were less demanding and that transparency was identified as a key factor for confidence independent of the tool used. However, the absence of a genuine increase in accuracy between both conditions raises questions and contradicts previous research. While this study’s findings demonstrate the potential for augmenting process mining tasks with GAI, they also highlight the need for further research into factors influencing confidence and ensuring reliable output. The lack of a genuine increase in accuracy when using LLMs, combined with a possible trend towards higher confidence at the same time, raises questions regarding the efficacy of such systems when used by novices in the PM context.