Integrating precision and power criteria in determining the number of raters needed for standard area diagram validation studies

Abstract Visual estimation of plant disease severity is widely used but inherently subjective, resulting in variability among raters. Standard area diagrams (SADs) are commonly employed to improve accuracy and precision; however, the number of raters required to validate SAD performance is often arbitrarily defined. This study aimed to establish an empirical framework to estimate the number of raters needed to achieve precise estimates of mean agreement and sufficient statistical power to detect improvements due to SAD use. Data from nine SAD validation studies across diverse plant disease pathosystems were analyzed. Agreement was quantified using Lin’s concordance correlation coefficient (LCCC). Two complementary criteria were applied: a precision criterion targeting a relative standard error of 0.10 for mean rater LCCC, and a power criterion evaluating absolute LCCC improvements of 0.05, 0.10, 0.15, and 0.20. Effects were assessed for feasibility relative to baseline agreement and converted exactly to the inverse hyperbolic tangent scale. The pooled precision estimate was seven raters, whereas study-specific precision estimates ranged from 2 to 16, indicating that the pooled value is not a universal recommendation. For an improvement of 0.10, combined sample-size estimates ranged from 5 to 32 raters across eight studies; this improvement was infeasible for pecan brown spot because baseline agreement was already 0.926. Required sample sizes depended on baseline agreement, variability in paired differences, and the specified improvement. These estimates provide context-specific guidance for SAD validation studies, although diagnostic and bootstrap analyses indicate that analytic power estimates require caution at small sample sizes. An interactive web application was developed to support adaptive estimation of rater numbers.

Authors

Institutions

Publication Details

Journal
European Journal of Plant Pathology
Published
2026-11-13
DOI
https://doi.org/10.1007/s10658-026-03339-7
Primary Topic
Reliability and Agreement in Measurement
Type
article
Field-Weighted Citation Impact
0.00

Funders

Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

Integrating precision and power criteria in determining the number of raters needed for standard area diagram validation studies

Ricardo Gomes Tomáz, Emerson M. Del Ponte, Clive Howard Bock, Mary Paz Romero-Benavides et al.
1 links
European Journal of Plant Pathology
Reliability and Agreement in Measurement
article

Integrating precision and power criteria in determining the number of raters needed for standard area diagram validation studies

Ricardo Gomes Tomáz, Emerson M. Del Ponte, Clive Howard Bock, Mary Paz Romero-Benavides, William C. Nunes
article en

Abstract

Abstract Visual estimation of plant disease severity is widely used but inherently subjective, resulting in variability among raters. Standard area diagrams (SADs) are commonly employed to improve accuracy and precision; however, the number of raters required to validate SAD performance is often arbitrarily defined. This study aimed to establish an empirical framework to estimate the number of raters needed to achieve precise estimates of mean agreement and sufficient statistical power to detect improvements due to SAD use. Data from nine SAD validation studies across diverse plant disease pathosystems were analyzed. Agreement was quantified using Lin’s concordance correlation coefficient (LCCC). Two complementary criteria were applied: a precision criterion targeting a relative standard error of 0.10 for mean rater LCCC, and a power criterion evaluating absolute LCCC improvements of 0.05, 0.10, 0.15, and 0.20. Effects were assessed for feasibility relative to baseline agreement and converted exactly to the inverse hyperbolic tangent scale. The pooled precision estimate was seven raters, whereas study-specific precision estimates ranged from 2 to 16, indicating that the pooled value is not a universal recommendation. For an improvement of 0.10, combined sample-size estimates ranged from 5 to 32 raters across eight studies; this improvement was infeasible for pecan brown spot because baseline agreement was already 0.926. Required sample sizes depended on baseline agreement, variability in paired differences, and the specified improvement. These estimates provide context-specific guidance for SAD validation studies, although diagnostic and bootstrap analyses indicate that analytic power estimates require caution at small sample sizes. An interactive web application was developed to support adaptive estimation of rater numbers.

European Journal of Plant Pathology
Universidade Estadual de Maringá (BR), Agricultural Research Service (US), United States Department of Agriculture (US), Universidade Federal de Viçosa (BR), U.S. Horticultural Research Laboratory (US)
U.S. Department of Agriculture, Universidade Federal de Viçosa, Coordenação de Aperfeiçoamento de Pessoal de Nível Superior, Conselho Nacional de Desenvolvimento Científico e Tecnológico, Agricultural Research Service
Reliability and Agreement in Measurement
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon