Promise and Caution for Composite Metrics for Quality Improvement in Neonatal Intensive Care

Structured quality improvement (QI) efforts have undoubtedly improved care and outcomes for high-risk newborns and their families.1 Many single-center and collaborative reports from neonatal intensive care units (NICUs) have documented significant and sustained improvements in process and outcome measures, particularly for very low birth weight (VLBW) infants.2,3 With the belief that QI works, robust approaches for safety and quality are now foundational elements of NICU care, as well as public health and regulatory frameworks that evaluate and monitor this care.In this issue of Pediatrics, Kaempf and colleagues present a report questioning whether QI efforts are continuing to improve outcomes for the highest-risk preterm infants.4 Using data for VLBW infants from 16 NICUs affiliated with their health system from 2010 to 2023, they examine risk-adjusted quality metrics combining 7 NICU morbidities and mortality. The authors combine multiple short-term outcomes into a single performance metric (“the Benefit Metric”) based on evidence linking more NICU morbidities to worse long-term outcomes, and they further examine this composite metric by hospital length of stay (“the Value Metric”) as an indicator of cost-effectiveness of care. Performance on these 2 metrics over this 14-year period showed no improvement, despite a strong QI culture across their system. The authors suggest that their results are similar to those reported by other, larger national and international networks, with limited evidence of improvement over the past 10 to 15 years, particularly for the smallest preterm infants.The authors offer numerous potential explanations for these trends, including limitations in QI experience, resource constraints, inadequate metric risk adjustment, and the biology and pathophysiology of extreme prematurity, and call for greater attention to all these areas. Although the relative importance of these various contributors may be debatable, we would all agree with dedicated efforts to identify barriers to further improvement in neonatal intensive care, and, undoubtedly, these efforts are widespread and ongoing. In the meantime, for those taking care of infants in the NICU, how should the results of this study inform ongoing local QI initiatives? For those focused on quality and safety, what do these results tell us about the value and potential limitations of structured QI more broadly?Further reflection on the composite metrics used may provide perspective on these questions. Although none would argue with the goal of reducing any NICU morbidity or with the goal of using resources optimally, this does not necessarily indicate that these metrics have utility for the evaluation of quality of care. Composite metrics are widely used in health care, with numerous proposed benefits arising from summarizing multiple dimensions of care into a single value.5,6 However, composite measures have limitations and risks, including less ability to assess specific improvement targets, more difficult risk adjustment, varying applicability of component measures based on center, and differences in preferred weighting of measures based on audience.7,8 Importantly, each component measure must be scientifically robust. The authors acknowledge some of these challenges, particularly risk adjustment; several other considerations may be worth highlighting.First, chronic lung disease (CLD) dominates the composite metric, occurring several times more commonly than the other outcomes, and CLD as a measure has limitations. The optimal definition of CLD remains elusive.9 The authors use the long-standing and common definition of oxygen need or respiratory support at 36 weeks postmenstrual age. There is increasing consensus that a severity-based definition better predicts outcomes in VLBW infants, with mild disease being less associated with adverse long-term outcomes.10–13 Among infants diagnosed with CLD using the traditional definition, the vast majority will have mild disease.14 In addition, although respiratory QI reports show potential for impact on CLD rates, these remain limited, and respiratory QI efforts may have more impact on other respiratory measures.15,16 The traditional definition of CLD may not be the best respiratory measure to assess respiratory-targeted improvement.Second, with its high relative frequency of CLD, the composite metric may mask important other improvements. The authors’ data suggest promising decreases in several morbidities, including any late infection, necrotizing enterocolitis, and retinopathy of prematurity, with risk-adjusted rates in some populations lower than expected in recent years. This matches reports from larger networks.17,18Third, mortality can complicate outcome measures, including length of stay. Mortality is typically considered a competing outcome for preterm infants, and survival can be associated with increases in rates of other morbidities.19,20 The authors appropriately consider mortality other than early deaths in the delivery room or initial hours of life in their composite metric and report no changes in this rate over their study period. However, this early mortality may have decreased over time, particularly for the youngest infants born at 22 to 23 weeks’ gestation, given national trends with increasing rates of resuscitation and intensive care.21 Changes in survival for this population could affect rates of other morbidities and could substantially affect average length of stay and, thereby, the composite value metric.Fourth, the equal weighting of the 7 morbidities in the composite metric is notable. Although the authors explain that the size of their data set limits their ability to explore other approaches to weighting, it is possible and even likely that providers and families would place greater importance on certain morbidities than others. There is also a notable disconnect between clinical diagnoses and outcomes deemed important by families, both in the NICU and in childhood.22–24 Using metrics based on existing databases allows for long-term analysis across multiple centers, an important strength of this work. Incorporating more nuanced preferences into defining and weighting the components could further strengthen the value of the composites.How can these considerations help us frame the work of Kaempf and colleagues? Their ambitious target of morbidity-free survival should be the goal of all who care for VLBW infants. Although their metrics may have limitations as an assessment of quality, several potential lessons can inform our ongoing QI efforts. First, this report and others confirm that QI initiatives continue to make significant improvements in certain morbidities for VLBW infants. Continued improvements in infection, necrotizing enterocolitis, retinopathy of prematurity, and growth are achievable and important. Second, we need to continue to focus on how best to measure respiratory outcomes in preterm infants, with a specific focus on quality measures. As the gestational age distribution of very preterm infants changes, current limitations to respiratory outcome measures may become even more challenging. Third, as we continue to collaborate with families to define outcomes that matter to them, we should explore further the benefits and challenges of incorporating family preferences into QI metrics.Finally, we should strive to ensure that the metrics that we use to assess quality of care are rigorous. Frameworks for evaluation of quality metrics include elements such as importance, scientific acceptability, including reliability and validity, feasibility, and usability.25,26 These frameworks are typically used for metrics for external assessment of performance, but many principles still apply to metrics used for local improvement.27 The authors of this report should be commended for decades of attention to their composite measure, and their work does begin to address some, although not all, of these framework elements.28,29 Ensuring that our measures are important, reliable, and valid should be one of our first steps in our QI efforts.