Increasing dependability of caregiver implementation fidelity estimates in early intervention: A generalizability and decision study

There is an increasing need to measure caregiver implementation of strategies from Naturalistic Developmental Behavioral Interventions (NDBIs) as a possible key mediator of outcomes in a child. The NDBI Fidelity (NDBI-Fi) rating scheme is a macro-code to estimate the implementation of core strategies. Yet, there is a need to understand the dependability of this measure to ensure intervention study findings are generalizable to everyday interactions and comparable across studies. We addressed this by evaluating the dependability or consistency of NDBI-Fi scores for 20 caregivers, averaged across observations of two occasions of two routines that were each scored by two raters. Our findings indicated that a single score (i.e. from a single occasion, single routine, and single rater) from the measure has low dependability (<i>g</i> = 0.43). When scores were averaged across two observations each of two routines scored by two raters (i.e. eight scores total), the score was more dependable (<i>g</i> = 0.77). The majority (81.6%) of absolute error variance was attributable to occasions of observation and its interaction with other facets (routine or rater). Therefore, we recommend the NDBI-Fi be applied to more than one observation of more than one routine to strengthen confidence that scores are generalizable to everyday parent-child interactions.Lay abstractOutcomes from caregiver-mediated interventions typically include measuring the caregiver's use of key techniques. The Naturalistic Developmental Behavioral Intervention-Fidelity (NDBI-Fi) tool is a valid measurement strategy for estimating caregiver use. In this study, we sought to understand how to improve data collection from natural observations of caregivers with their children to ensure the scores are representative of how the caregiver and child typically interact. We observed 20 caregiver-child pairs via telehealth in snack and play routines over two different days. Each video was rated using the NDBI-Fi by two observers. We learned that increasing the number of observations may be the best way to improve the dependability of scores from natural caregiver-child observations. This study adds to recent research seeking to understand how to best measure caregiver strategy use. These findings may guide future researchers and clinicians to consider increasing the number of observations used to evaluate caregiver use of intervention techniques in research studies or clinical practice.