Enhancing Cognitive Accessibility in Mobile Apps via LLM-driven Redundant Entry Reduction

Currently, mobile applications (apps) have become indispensable in our daily lives, supporting activities such as online shopping, scheduling, and health management. However, users frequently encounter redundant information entries, where identical data like dates, addresses, or preferences must be repeatedly entered across multiple Graphical User Interfaces (GUIs) or sessions. This problem not only increases interaction effort and user frustration but also disproportionately affects individuals with cognitive impairments, who may experience difficulty remembering or managing redundant entries. Traditionally, developers often rely on static field mappings and predefined rules to automatically fill such redundant entries, which are frequently inflexible, failing to adapt to diverse app layouts, dynamic interface elements, or individual user needs. To address this challenge, we propose LLM-DIP, the first large language model (LLM)-driven framework that automatically detects and fills non-sensitive redundant entries by leveraging a multimodal understanding of GUI screenshots and layout structures. We evaluated LLM-DIP on 100 Android apps across 17 domains, achieving an average accuracy of 91.6% and F1-score of 91.0%, which outperforms the existing baselines. Additionally, a user study involving 20 participants with cognitive impairments confirmed its practical benefits, demonstrating increased input efficiency and high user satisfaction (average rating: 4.3/5).