SPOC: A Scale for Potential Operation Consequences of UI Interactions

Many approaches rely on a semantic understanding of screen contents to perform further actions on this information. This is relevant for test automation, robotic process automation, or context based assistive tasks (digital agents). Most of these approaches just analyze basic information about user interface elements but fall short in predicting the consequences of single user interface actions. Possible consequences of UI actions include (permanent) data loss, effects on the reputation of people (e.g. posting on social media), or other irreversible consequences. This makes it necessary to identify and analyze the potential consequences of operation on user interfaces to prevent automatic systems from making possibly fatal mistakes with a high impact on the human user. This paper presents a Scale for Potential Operation Consequences (SPOC) and demonstrates an automatic rating system based on Large Language Models (LLMs), that can be performed locally or in cloud implementations. The LLM-based ratings were tested against a human control group based on a questionnaire (n=35), and proved to perform as well as their human counterparts, making SPOC a robust and automation-friendly rating system for user interface element actions.