User Perceptions of Personalized and Generic Explanations in LLM-Driven Recommender Systems

The adoption of recommender systems (RSs) in various domains has become increasingly popular, but concerns have been raised about their lack of transparency and interpretability. While significant advancements have been made in creating explainable RSs, there is still a shortage of automated approaches that can deliver meaningful and contextual human-centered explanations. Numerous studies have evaluated explanations based on human-generated recommendations and explanations to address this gap. However, such approaches do not scale for real-world systems. Building on recent research that exploits Large Language Models (LLMs) for RSs, we propose leveraging the conversational capabilities of ChatGPT to provide users with personalized, human-like, and meaningful explanations for recommended items. Our paper presents a user study with 94 participants that measures users’ perceptions of ChatGPT-generated explanations when acting as a recommender system. Regarding recommendations, we assess whether users prefer ChatGPT over random (but popular) recommendations. Concerning explanations, we assess users’ perceptions of personalization, effectiveness, and persuasiveness. We also break down the explanations in its constituent arguments and investigate the differences in argument types between generic and user-specific explanations. Our results show that participants rated ChatGPT’s recommendations significantly higher than random ones ( \(\beta=-.53\) , \(p<.001\) ). Surprisingly, user-based explanations that explicitly referenced participants’ preferences were not perceived as more personalized or persuasive than generic ones, except when the recommended movie was unfamiliar, where user-based explanations became more effective ( \(\beta=0.35\) , \(p<.05\) ). Overall, our findings highlight both the promise and the limitations of using LLMs for generating personalized explanations, suggesting that personalization is most impactful when users lack prior knowledge of the item.