Evaluator-associated variation in routine postoperative activity-pain NRS scores: a retrospective observational study of 126,112 assessments

The Numeric Rating Scale (NRS) is widely used for postoperative pain assessment, but direct evidence on how comparably scores are elicited by different evaluators in routine postoperative care remains limited. Whether observations made within variable clinical workflows can be interpreted in the same way as controlled duplicate ratings remains uncertain. We analysed 126,112 routine postoperative activity-pain assessments (NRS 0–10) from 45,177 patients evaluated by 15 anaesthesia nurses at a single tertiary hospital between 2020 and 2025. Evaluator-associated variation was quantified using cross-classified linear and logistic mixed-effects models with patient and evaluator random intercepts. Evaluator identity accounted for 33.3% of activity-pain NRS variance (95% CI 17.4–47.6%), comparable to the patient component (33.0%). One evaluator with a markedly restricted scoring pattern (NRS 1–3 only) was disproportionately influential. After exclusion of this evaluator, the evaluator variance partition coefficient remained 24.1% (95% CI 10.9–37.3%), and the span of model-derived evaluator contrasts remained 1.82 NRS points. Within-patient, bridging-subset, extended case-mix, year-restricted, ordinal, and threshold sensitivity analyses were compatible with evaluator-associated variation, although the magnitude varied across specifications. In this routine-care observational dataset, evaluator identity was associated with variation in activity-pain NRS scores. The estimates may also capture differences in non-random evaluator assignment, workflow, and assessment timing. The study identifies heterogeneity in recorded scores but does not establish effects on clinical decisions or patient outcomes.