The ErrVarNorm (EVN) index in Case 1 Best–Worst Scaling: Further methodological and data quality investigations

The error variance normalised (ErrVarNorm; EVN) index, introduced by Llobell et al. (2025), measures participant response consistency in Case 1 Best–Worst Scaling (BWS) tasks. In the context of food-related consumer research, this methodology-focused study considered whether EVN was influenced by participants' country of residence (Obj. 1) and its relationship to other data quality metrics and indices (Obj. 2). Responses from over 9500 participants across nine surveys were analysed. Study 1 ( N = 4122) found lower response consistency among participants from China and India than from New Zealand and the UK, while Study 2 ( N = 1084) showed similar performance between Germany and Singapore, suggesting that factors other than country of residence underpinned the observed differences. Supporting this, within-country variation in Study 3 ( N = 4546) was similar in magnitude to between-country variation in Study 1. Participants failing standard quality checks had markedly lower EVN values: speeders averaged 0.18–0.32, compared with 0.71–0.76 for those who were not speeders; flatliners 0.36–0.56 vs. 0.70–0.77, and those failing instructional manipulation checks 0.39–0.45 vs. 0.79–0.83. Study-specific checks yielded similar patterns, with poor-quality open-ended responses averaging 0.42 vs. 0.77 and contradictory answering averaging 0.40 vs. 0.71. Combining multiple checks further increased the mean EVN of retained participants but reduced sample size, highlighting the need to balance quality thresholds with the number of consumers retained for analysis. Given its consistent association with multiple quality indicators, EVN appears to be a valuable proxy for individual-level data quality beyond Case 1 BWS. • Provides insights on EVN index for measuring respondent consistency in Case 1 BWS. • Within-country EVN variation matched between-country differences in magnitude. • Participants failing quality checks (speeding, flatlining, attention) had systematically lower EVN. • EVN serves as an individual-level overall data quality indicator in online surveys.