CONTEST: A generalization of ONEST to estimate sample size for predictive augmented intelligence method validation studies

Laboratories must verify and validate assays before reporting results in the clinical record. With the advent of machine learning algorithms, multiclass decision-support tools are coming online but the FDA explicitly does not contemplate multiclass problems in their guidance for test validation. Validation requires, for a laboratory's patient population, evaluation of four performance characteristics to a reference method: accuracy, precision, reportable range, and reference intervals. In the absence of a reference method, proportion of agreement is the appropriate metric (Meier 2007). For subjective tests, the traditional metrics for precision are in the area of interrater reliability, and interrater reliability is well studied in the pathology literature ("Gwet Handbook of Interrater Reliability 4th Ed.pdf," n.d.). Recently, Guo and Han introduced an alternative framing, Observers Needed to Evaluate a Subjective Test (ONEST). This article introduces a treatment effect extension of ONEST, Cases and Observers Needed to Evaluate a Subjective Test (CONTEST) and demonstrates that the agreement and disagreement distributions can be reasonably specified with parametric probability distributions such that the required sample size for a test, at a given level and power, can be calculated. We argue that this would be an appropriate method to develop for validation of tools used to augment a subjective test, given a prior set of cases, observers, and decisions, such as from another archive, cohort, or dataset, particularly in resource-constrained settings.