Sentiment Analysis of Government Mobile App Reviews: Validating Zero-Shot Classification with a Large Language Model

Abstract User reviews of mobile applications provide a source of citizen feedback on digital public services, yet their automated analysis in non-English settings is constrained by the limited availability of domain-adapted classifiers. This study develops and evaluates a documented procedure for three-class sentiment classification of Czech reviews of government mobile applications using a large language model without task-specific fine-tuning. The corpus comprised 10,198 reviews of six applications. An adjudicated human reference was created for a stratified random sample of 500 reviews, and LLM performance was compared with a majority-class baseline, two lexicon-based methods, and two readily deployable encoder-based transfer models. The validation included five repeated runs under identical settings and an analysis of classification errors. Across five GPT-5.6 Sol validation runs, mean accuracy was 93.0%, mean macro-F1 was 0.827, and mean Cohen's kappa was 0.857. Macro-F1 ranged from 0.814 to 0.845, while agreement among the five runs was high (Fleiss' kappa = 0.918). The neutral class was the most challenging, with most disagreements occurring at its boundaries with negative and positive sentiment. In the full corpus, 65.9% of reviews were classified as negative, 7.9% as neutral, and 26.2% as positive. Textual sentiment was strongly associated with star ratings but was not identical to them. These findings provide corpus-specific evidence that a standardized LLM procedure can help analyze short-form non-english citizen feedback. They also underscore the need for human reference labels, stability checks, and precise documentation of the model, prompt, settings, and execution workflow.