Mobile Application Security Assessment through Large Language Models (LLMs) based on User Reviews

Digital payments via mobile UPI (Unified Payments Interface) applications have recently become increasingly popular due to their ease of use. However, the users face significant security risks during the usage of the mobile applications. Hence security analysis based on user reviews gains importance. The existing literature focuses on bugs, confidentiality and usability issues providing partial insights into application security. In this paper, we present a systematic approach to assess the security of the mobile UPI applications based on user reviews with respect to eleven security goals given by GoI. We model the problem as a Multi-Label Text Categorization problem in which each review is labelled with a subset of the security goals. We developed a zero-shot prompt-based method using Large Language Models (LLMs), which do not need extensive labelled data, to categorize the user reviews. By leveraging expert knowledge and employing group consensus method, we have created an extremely useful annotated dataset of 4,000 mobile application user reviews for ground truth validation. The proposed Multi-Label Text Categorization approach is tested on three datasets including ours and two public datasets from different domains like News and Medical articles. Our approach outperforms the other deep learning methods like SBERT, ROBERTa etc. by showing 14.6% improvement on the best result, on average, with respect to F1 score and 34.1% improvement on Hamming loss. We have collected approximately 1.7 million user reviews on 30 UPI applications from the Google Play Store. We conduct qualitative analysis to demonstrate our approach and offer actionable recommendations for developers with respect to app security.