DesignWatch: Analyzing Users' Operations of Mobile Apps Based on Screen Recordings

Screen recordings of users’ operations to complete tasks in the mobile app are vital resources for designers to assess the app’s usability. However, analyzing these recordings at a large scale could be mentally challenging. In this paper, we present DesignWatch, which assists designers in analyzing users’ operations of mobile apps based on collected screen recordings. DesignWatch supports interactive visual analyses of multiple users’ operation paths in the app and prompts GPT-4 with vision to simulate users’ thoughts during each operation. We conduct expert interviews with four designers, which highlight DesignWatch’s usefulness in helping them quickly understand users’ operation patterns in the app, identify the potentially problematic UI design page, and get insights for improving the app design. We conclude with design implications for facilitating usability tests with interactive visualization and generative models.