User perceptions of serendipity-control, cognitive-level labelling, and explanation features in an LLM-based academic article recommender: a single-institution pilot study

Serendipity in recommender systems has been studied mainly as an algorithmic property rather than as an interface-design question. This pilot study examined how undergraduates perceived three interface features of ScholarMind, a prototype in which a large language model (LLM) generates article recommendations directly: accuracy and serendipity sliders, LLM-assigned Bloom’s taxonomy labels, and an explanation panel of LLM-generated weights labelled as SHAP values. Forty-one students at one institution in Pakistan completed a task requiring use of all three features, followed by a self-report battery (ResQue, SUS, UEQ-S and three unvalidated feature scales). Mean feature ratings lay on the agreement side of the midpoint (3.62–3.80 on 5-point scales), but reverse-worded items correlated negatively with their scales, indicating acquiescent responding. Exploratory correlations linked explanation-panel perceptions with perceived trust (rₛ = 0.47, 95% CI [0.18, 0.68]) and Bloom’s-label perceptions with perceived usefulness (rₛ = 0.51, 95% CI [0.23, 0.71]). Perceived usability was below benchmark (SUS = 56.6 vs. 68), and preference relative to Google Scholar was evenly split. Inspection of the source code showed that the sliders only reorder a fixed set of generated articles. The findings describe perceptions of an interface prototype, not recommender performance.