Evaluating user-facing risk communication in online scam-checking services: the consumer warning quality index (CWQI)

Purpose Scam-checking services present scores, labels, colours, explanations and advice that may influence whether users proceed with a website. This paper aims to propose a structured framework for evaluating these outputs as user-facing security warnings. Design/methodology/approach The consumer warning quality index (CWQI) covers clarity, specificity, evidence, actionability, education and visual communication. It was applied to 180 outputs from six services across 30 website cases (27 source-confirmed risky sites and three legitimate controls). Alignment was coded separately. Inter-coder reliability was assessed for the six CWQI dimensions on a stratified 20% sample (36 outputs) and for alignment on a stratified 48-output subset. Findings Communication quality differed significantly across services (χ2(5) = 59.11, p < 0.001, W = 0.39). In the primary coding, clarity and visual communication had the highest means; actionability and education had the lowest, although fine-grained dimension rankings require caution where reliability was lower. Under-warning (18.3%) and false reassurance (10.0%) affected 28.3% of outputs (21.7–35.0% across the alternative score-threshold scenarios). CWQI and severity-weighted alignment were moderately associated (Spearman ρ = 0.50, p < 0.001); false-reassurance cases nevertheless included prominently presented reassuring judgements, illustrating that clear presentation can accompany an inappropriate risk level. Practical implications CWQI provides a structured and transparent coding approach for identifying communication weaknesses in archived outputs without access to proprietary detection models, subject to further validation. Originality/value CWQI provides an exploratory, theory-informed framework for operationalising six communication characteristics of scam-checker outputs. Its principal contribution is to support systematic cross-service comparison while keeping communication quality analytically separate from alignment with source-confirmed risk.