Divergent user expectations of conversational AI: a comparative sentiment and thematic analysis of ChatGPT and Replika app store reviews

Abstract Conversational AI apps are increasingly serving two different purposes: instrumental productivity and emotional support. Although both are built on the same underlying language technology, users may evaluate these systems according to different instrumental and relational expectations. This study examines the differences in user evaluation of a task-oriented assistant (ChatGPT) and a companionship-oriented agent (Replika) through computational analysis of the sentiment scoring, NRC emotion analysis, topic modelling, and inferential tests of the 6,000 publicly available app store reviews (3,000 per application). The two applications differed only marginally in overall review polarity, and the direction of the difference depended on the method used: under the Bing lexicon Replika reviews scored slightly higher (M = 0.312 vs. 0.275), whereas under sentimentr ChatGPT scored higher (M = 0.135 vs. 0.102); aggregate polarity was therefore not a robust point of difference between the two corpora. The rating distributions differed significantly between the applications (χ² (4, N = 6,000) = 242.26, p < .001, Cramér’s V = 0.20), with ChatGPT ratings more polarised at both ends of the scale (36.8% vs. 21.7% five-star; 27.4% vs. 23.8% one-star) and Replika ratings more concentrated in the intermediate categories. Once scores were normalised for review length, Replika reviews contained more joy-, anger-, surprise-, and anticipation-associated language, with all effect sizes small; fear-, sadness-, and trust-associated language did not differ significantly between the applications. The coupling between review language and star rating was weaker for the companion application, though the difference was small (App × Rating interaction, partial η² = 0.016); because the two samples span different publication periods, this interaction is reported as provisional. Topic modelling yielded a six-topic solution in which conversation and companionship accounted for 53.6% of Replika reviews against 10.9% of ChatGPT reviews, while task assistance and usefulness accounted for 30.9% of ChatGPT reviews against 1.2% of Replika reviews. Together, the findings are consistent with differences in evaluative framing between the two applications examined in this study, illustrating two representative cases rather than establishing distinct categories of conversational AI systems.