Moving Predictive Analytics From Populations to Individual Children With Medical Complexity

In this issue of Pediatrics, Cohen and colleagues1 present a method to classify children with medical complexity (CMC) in a way that advances our understanding of patterns of health care use in the CMC population and hopefully will eventually enable effective customization of services for individual patients “at different risk levels for persistent high health care use.”This represents a noble, noteworthy, and challenging undertaking. Ample evidence has documented that CMC, who account for a large and increasing proportion of health care use and expenditures in hospital2 and ambulatory settings,3,4 experience poor health outcomes5 and have substantial unmet needs,6,7 whereas their families bear excessive care and care coordination duties8 and grapple with significant financial challenges.9,10At the same time, based on both published data and decades of extensive clinical experience, we know that the CMC population varies greatly in the severity and fragility of their medical state and in their, along with their families’, needs.11–13 This phenotypic variation has long challenged classification systems for children with special health care needs generally and CMC in particular, with approaches tending to emphasize either diagnoses and procedures14–17 or “functional” information18,19 (which might be direct measures or reports of a given child’s functional status or measures of past and current health care use that serves essentially as a surrogate measure of their physiological or medical stability “function”).Cohen and colleagues combined both approaches, as they used the statistical method of latent class analysis with variables regarding patients’ demographic and clinical characteristics, as well as health care use (ie, emergency department visits, hospitalization, and home health care) in 2017. The authors ultimately identified 3 subgroups of CMC, which we can call “+++” (ie, high levels of neuro-disability, high levels of technology dependency, and high levels of multimorbidity, comprising 9.1% of all CMC patients), “+–” (ie, high levels of neuro-disability and low levels for the other 2 characteristics, comprising 14.8% of CMC patients) and “—” (low levels for all 3 characteristics, comprising 76.0% of CMC patients). They then examined the subsequent health care use and costs for these 3 groups in 2018 and 2019: the “+++” group, on average, had the highest health care use and costs, the “—” group had the lowest, and the “+–” group was in the middle.This set of findings provides 2 important insights regarding the patterns of health care use in the overall population of CMC. First, most are not fated to experience extremely high health care use, whereas a minority are. Second, neuro-disability, technology dependencies, and multimorbidity appear to be risk factors. The authors conclude that this approach could be used in future research “for risk-stratification and prognostication in outcome studies.” They go further and state that CMC “can be grouped into unique classes that can usefully predict future health care use patterns, with potential utility for aligning clinical programs and other interventions to address their needs.”The task now is to address the vitally important difference between the accuracy of the groups identified in this study (let us call this group-level accuracy) and the accuracy when applying these group-level categories to individuals within each group (individual-level accuracy). Notably, a classification system may have outstanding performance at the group level but have much poorer performance at the individual level (a phenomenon repeatedly observed in studies that we have conducted).20–22Indeed, living as we now do in an age of predictive analytics and targeted therapies, spurred on by artificial intelligence (AI) and big data, many analytical projects make this move from populations to distinct people but do not reflect the resulting importance of assessing the accuracy of individual-level predictions and concerns regarding benefits, harms, and equity.23–25First, the initial priority when evaluating an individual-level classification system is to conduct a thorough assessment of the accuracy of individual-level predictions. Reporting a summary metric of the overall performance of a statistical model, an AI algorithm, or a classification system with criteria such as the area under the receiver operating characteristic curve or the C statistic, or the Akaike or the Bayesian Information Criteria reveals little about how well the model, algorithm, or system will perform at the individual level (the area under the precision-recall curve is much better). Instead, we evaluate individual-level classification accuracy using the sensitivity, specificity, positive and negative predictive values, and other so-called test characteristics of medical diagnostic and diagnostic screening tests, which classify individual patients as either being at high or low risk of having a disease or condition. These same analytical concepts apply to prognostic classification and treatment decision systems. If the test characteristics are poor, classifications and subsequent decisions should be judged to be unacceptably inaccurate for implementation.Second, the test characteristics of outcome measures must be assessed in the form that these outcome measures will be used. For natively dichotomous outcomes such as mortality (with death compared with alive), the measure is already in the form in which it will be used. For most other outcome measures, however, the measures represent a range of values but often are dichotomized when put into use. For example, health care expenditures range from none to large amounts of money. An outcome model of health care expenditures may fit optimally if the expenditures are dichotomized to separate none or very low expenditures from all higher expenditures, but the desired use of the measure in a hypothetical health care management program may be to identify patients above the 90th percentile of health care expenditures in the population (a much more difficult task). This second form of the outcome measure is what needs to be tested.Third, we must check for “conditional” or “local” group or subgroup effects. For example, most individual-level test characteristics are highly dependent on the prevalence of the outcome that is being predicted, and accurately predicting the occurrence of an unlikely outcome is inherently more difficult than predicting a likely outcome. Because prevalence of outcomes varies across populations, across subgroups within a population, or across sites of care, each prediction system must be evaluated in the population and setting where the system will be used.Fourth, we must check for embedded bias within the data that are being used to generate the predictions.26 Associations between patients and outcomes can be caused by the intrinsic nature of their underlying conditions or potential structural biases such as differential access to health care or poverty resulting in poorer health status. Predictions that propagate these structural biases forward are deeply problematic.Fifth, we must engage in comprehensive ethical analysis of both the predictions (requiring empirical analysis, transparently reported, as outlined earlier, regarding their accuracy overall and across relevant subgroups and their potential biases) and of how the predictions will be used.23 For example, if the predictions will be used to select patients to enroll into a health care management program, we should ask the following questions: How will the patients enrolled in the program benefit? What costs or burdens or tradeoffs will the program impose on the enrollees? Are there any potential harms stemming from the program? How fair and unbiased is the selection for those not enrolled?With the rise of “big data,” AI technologies, and predictive analytics, this set of steps represents an urgent and imperative agenda, not only for CMC but for all our patients and, indeed, for all of us.