Simple formula for adjusted performance parameters in evaluation of diagnostic methods

In medical science, diagnostic methods are validated and evaluated using certain diagnostic performance parameters such as accuracy, sensitivity, specificity, positive predictive value, and negative predictive value,1, 2 see Box 1. In the presence of a reference method, these performance characteristics can easily be calculated to evaluate the performance of a new method. Such performance characteristics are also widely used for comparing one method with another method or to evaluate the performance of a methodology, for example, through comparative analysis such as systematical review or meta-analysis.3 Accuracy = (TN + TP)/(TN + TP + FN + FP) *100. Sensitivity = TP/(TP + FN) *100. Specificity = TN/(TN + FP) *100. Positive Predictive Value = TP/(TP + FP) *100. Negative Predictive Value = TN/(TN + FN) *100. TP = True Positives. TN = True Negatives. FP = False Positives. FN = False Negatives. A frequently encountered problem is that existing performance parameters (Box 1) do not take sample size into account. This has two implications: (1) An evaluation of a method showing 100% accuracy based on a small number of samples may be poorly substantiated, overrated, or even misleading; and (2) a comparison of different methods becomes misleading or erroneous when the reported parameters rest on calculations from widely different sample sizes. In addition, existing measures of uncertainty, such as standard error, coefficient of variation, and confidence intervals, need to be interpreted in conjunction with the performance parameter, resulting in a complex performance evaluation. A confidence interval, which does take the sample size into account, offers some help for evaluating the precision of an accuracy parameter. By definition, the confidence interval provides an interval, in which, by a certain likelihood, a repeated measurement would fall into. The lower limit will indicate a boundary at which it would be unlikely to fall below. However, the lower limit is not a description of the actual performance parameter, and the confidence interval in itself does not present a single metric, which can easily be used for method evaluation or comparison. We propose a simple solution introducing adjusted performance parameters, as presented in Box 2, in which the estimated performance and the uncertainty in that estimate is incorporated into a single operational metric. We present the argumentation for using this approach and provide illustrative examples. Finally, we reflect on the options for studies with small sample sizes. Accuracy > (TN + TP-1)/(TN + TP + FN + FP) *100. Sensitivity > (TP-1)/(TP + FN) *100. Specificity > (TN-1)/(TN + FP) *100. Positive Predictive Value > (TP-1)/(TP + FP) *100. Negative Predictive Value > (TN-1)/(TN + FN) *100. TP = True Positives. TN = True Negatives. FP = False Positives. FN = False Negatives. First, consider a couple of imaginary examples. If a person points at a pregnant woman and guesses that her fetus is a girl and it turns out to be correct, we can safely say that this person's prediction method is 100% accurate by existing theory. But somehow, we do not necessarily trust that this person really knows how to tell whether a pregnant woman carries a male or female fetus. Why is that? The obvious reason is the sample size. Had this person made the correct guess in 50 out of 50 cases, it would have been quite impressive. The sample size is thus a key part of the value behind a measured performance parameter of any method. This has implications for the reliability of a method in the general sense that the larger the sample size, the more reliable the assessment of the method's accuracy. The larger a sample size, the more representative the data become of the true performance of the method. And further, the larger a sample size, the less the uncertainty. However, sample size also has implications for comparing two or more methods. When comparing two methods, we could ask: which method would be the best to implement? Method A with an accuracy of 100% (n = 50) or Method B with an accuracy of 98.5% (n = 1000)? Obviously, Method A has the highest accuracy. However, Method A's accuracy is substantiated with a sample size of only 50 compared with Method B's sample size of 1000. Nevertheless, it could be that Method A's accuracy will remain 100% if tested on a larger sample size and thus always be better than Method B. Method A's future accuracy is indicated by the width of the confidence interval—or, specifically in this context, the lower limit of the confidence interval. In Figure 1A, we have shown how the 95% confidence interval relates to 100% accuracy with different sample sizes. Figure 1A thus shows how unreliable 100% accuracy would be at lower sample sizes, as depicted by the lower limit of the 95% confidence interval. The point is that the smaller the sample size, the more misleading or meaningless it may seem to express the accuracy as 100%. But in a logical context, what can we then say for certain about the performance of Method A? What we can say for certain is that, in any case, the demonstrated accuracy for Method A is better than if there had been one erroneous result (i.e., one erroneous result additional to the existing number of erroneous results). In other words, 50 correct results out of 50 tests are really only better than 49 results correct out of 50 tests. Logically, we can state that the accuracy of 50/50 correct outcomes is higher than the accuracy of 49/50 correct outcomes, which is 98%. So, in this logical context, the accuracy of Method A is with certainty higher than 98%, which we can express as >98%. If we do the same calculation for Method B, we can state again that the accuracy is better than if there had been one additional erroneous result. One additional erroneous result would thus give 984 correct results out of 1000 tests. So, the accuracy of Method B would be >98.4%. These adjusted accuracies are now equally affected by the sample size. Or more precisely, the accuracies are now adjusted to reflect the accuracies as substantiated by the number of tests used in the method evaluation. And in this example, Method B's adjusted accuracy is slightly higher than Method A's adjusted accuracy. Figure 1B shows how an adjusted accuracy can reflect the uncertainty seen in Figure 1A, as the sample size becomes incorporated into the estimation. With the adjusted accuracy, it becomes more obvious that a low sample size generates low accuracy despite 100% correct results. Essentially, we are probing the statistical performance parameters in the two situations for robustness in relation to sample size. Also, the adjusted accuracy situates itself centered between the limits of the confidence intervals (Figure 1B), thus acting as an impartial candidate for representing the data. Interestingly, it is not uncommon to observe one method with lower accuracy in a large sample size and another method with higher accuracy in a smaller sample size. In one review, for example, the average sample size was n = 694 for studies with 100% sensitivity, whereas the average sample size was n = 4696 for studies with less than 100% sensitivity.4 In another meta-analysis, the average sample size was n = 78 for studies with 100% accuracy, whereas it was n = 327 for studies with less than 100% accuracy.3 And in yet another study, the average sample size was n = 93 for studies with 100% accuracy, whereas it was n = 372 for studies with less than 100% accuracy.5 In general, if you encounter a couple of mistakes per 1000 samples tested, then it is always easier to get 100% correct results with a small sample size. However, this does not necessarily imply that the performance of the method with a small sample size is better. We suggest that our simple approach may enable an improved and statistically more feasible comparison of studies with 100% correct results using an operational parameter that takes the sample size into account. To take the sample size into account, we thus propose adjusting the existing performance parameters (Box 1) such that the adjusted parameter is written as higher than (>) the estimated performance had there been one additional erroneous result, which would effectively be simply to subtract 1 from the number of true results (as in: n-1/n). See the detailed calculations in Box 2 for different adjusted performance parameters. This adjusted approach has been used elsewhere in a review of studies, adjusting 100% correct results as proposed here.6 In this review, it was mentioned that such 100% correct results were reported differently to allow the reader to estimate the statistical power of each study.6 Our purpose of introducing adjusted performance parameters is exactly reflected in this statement. Examining examples of real data, we have looked at data that we published earlier from our laboratory in Copenhagen.7-9 These data were from a method for predicting the fetal RhD type by analyzing cell-free DNA in plasma from pregnant RhD negative women. With this analysis, high sensitivity is essential, and although the performances are all acceptable, they are exemplary for the aspect we discuss in this paper. In Table 1, we have presented the sensitivities reported from these three studies, and we also present the adjusted sensitivities as recalculated using the formula for adjusted sensitivity in Box 2. In Table 1, there is a clear difference in the progress for the reported sensitivities and the adjusted sensitivities. The reported sensitivities are 100% in the first two studies and then decrease slightly due to one false-negative result, regardless of the sample sizes. In the adjusted sensitivities, there is a slight increase from Studies I to III in sensitivity due to an increase in sample size, despite the occurrence of one false-negative result in the latter study. In Figure 1C, we have presented the adjusted sensitivities from the Copenhagen (CPH) data (as taken from Table 1) related to sample size. As the reliability in the performance is documented by increasing sample sizes, the adjusted sensitivities from the CPH data reveal a performance progress that is nonobvious in the traditional performance parameters. In relation to noninvasive fetal RHD genotyping, as exemplified in Table 1, the international recommendation for performance evaluation is to attain a 95% confidence interval where the lower limit for sensitivity should be ≥99%.10 Importantly, this criterion is only attainable with a sample size of at least 366 positive cases (with all cases correct). To further illustrate this issue, we depicted the required sample sizes needed to meet various criteria for lower 95% CI limits, Figure 2. The mentioned international recommendation, as well as the data in Figure 2 exemplifies the importance of obtaining a sufficiently large sample size. Moreover, it underscores that small sample-size studies estimating performance parameters such as sensitivity, specificity, etc., of a given assay must be interpreted with caution, as discussed below. In biology and medicine, analysis and diagnostics of rare conditions are often encountered. In these instances, obtaining a large series of test results can be challenging or even impracticable. The formulas for adjusted performance parameters may be used for such small sample size studies to present a more cautious assessment of the reliability of an analysis. It provides a tool to avoid overestimating assay performance. Using fetal blood group antigen genotyping as further examples, in this field several studies with small sample sizes have presented accuracies of 100%.11 Our group has presented such studies, with 100% accuracy for predicting fetal KEL1 (n = 2)12; ABO (n = 19)13; RHC (n = 5), RHc (n = 17), RHE (n = 8)14; and KEL1 (n = 34).15 Even though some of these studies are proof-of-principle studies, they exemplify the substantial uncertainties with smaller sample sizes. Adjusting the accuracies has significant consequences as to the presentation of the results, rendering adjusted accuracies of >50%, >94.7%, >80%, >94.1%, >87.5%, and >97.1%, respectively. Nevertheless, this may be a more appropriate way, statistically, to express the experience with the method to relevant clinicians. In any case, it provides an operational figure for a practical assessment of assay reliability in contrast to the more abstract recommendation of interpreting performance data with great care. With small sample size studies, we suggest that such adjusted performance parameters should at least supplement the classical performance parameters. Furthermore, these figures indicate that with a sample size of n = 20, an adjusted accuracy of more than 95% can be reached with our approach. This could be considered as a potential threshold for a recommendable minimal sample size in such studies where case samples are rare. We propose a simple adjustment to the classical method performance parameters of accuracy, sensitivity, specificity, positive predictive value, and negative predictive value. We propose that the adjusted performance parameters give a more reliable evaluation of the performance of a method by taking into account how sample size contributes to the value and documentation of the performance parameters. This is of particular interest in studies with 100% accuracy. The proposed adjusted performance parameters can be used as single metrics incorporating both the performance parameter and its uncertainty, allowing for a quick assessment approach in comparison with existing uncertainty measurements. Furthermore, the adjusted performance parameters allow for a better comparison of the performance from different studies, as done, for example, in meta-analysis studies. Estimating adjusted performance data also allows for a more realistic comparison with the existing performance presented in the literature. Finally, adjusted performance parameters allow for a more reliable assessment of the value of a method from studies with small sample sizes, avoiding overestimating the performance of a method. The authors have disclosed no conflicts of interest.