Quantifying consistency among interpreters of reference data for estimation of harvest area through visual interpretation of remote sensing data

Abstract Visual interpretation of remote sensing data is a fundamental procedure for collecting reference data for the accuracy assessment and area estimation of forest disturbance map products. However, variability among different interpreters can lead to biased estimates of both variance and area. This study aimed to quantify interpreter consistency in reference data for harvest area estimation and assess the relative influence of interpreter variability on total variance estimation using reference data obtained from multiple interpreters. A total of 31 000 sample units were collected through stratified random sampling, with each unit independently interpreted by three interpreters. A subset of 4000 sample units was interpreted by all six interpreters. The complete agreement among interpreters was 99.8% for the stable forest stratum, 84.0% for the harvest stratum, and 78.3% for the spatial buffer surrounding the harvest stratum. A variance estimator based on a simple measurement model showed that the standard error of harvest area estimation from a single interpreter could be underestimated by a factor of 1.2–1.8 for sample sizes from 10 000 to 31 000 if interpreter variability was not considered. The relative influence of interpreter variability on the total variance increased with sample size. When the majority of interpretations from three or five interpreters were used, interpreter variability was substantially reduced, suggesting that at least three interpreters per sample unit were required to generate consistent reference data. In contrast, for smaller sample sizes (e.g. n < 5000), the use of a single interpreter resulted in only a limited underestimation of the standard error (a factor of ~1.1). This study provides practical guidance for improving reference data collection in remote sensing–based visual interpretation workflows.