Psychometric Evaluation of the Modified Neonatal Resuscitation Program Adherence Assessment Tool When Utilized for In Situ Simulation and Telesimulation Scenarios

Neonatal resuscitation is a high acuity, low occurrence event that requires precision, teamwork, and rapid decision-making. Simulation using Neonatal Resuscitation Program (NRP) guidelines allows teams to build competency and preparedness, improving neonatal outcomes. Accurate, reliable performance evaluation during simulation is essential to the provision of meaningful feedback. Evidence supporting the reliability and validity of existing tools when used in telesimulation (TS) is limited. We sought to evaluate the psychometric properties of the modified NRP (mNRP) tool when assessing interprofessional team performance in onsite and TS environments.We employed a methodological design to conduct secondary analyses of data from a larger study using simulation to enhance resuscitation skills as assessed by the mNRP. Item-level data from 96 simulations were subjected to Classical Test Theory-based item analyses, including evaluation of item difficulty, discrimination, and item-to-total correlation; reliability; and exploratory factor analysis (EFA). Interrater reliability (IRR) between novice and expert raters was assessed for a subset of cases.Item difficulties reflected a desirable mix of difficulty in endorsement, suggesting that items capture a range of guideline adherence. Most item discrimination (range: 0.05-0.81) and corrected item-total correlation (range: 0.005-0.68) values had moderate-to-strong, positive correlations with total scores, indicating discriminative ability. EFA yielded three and four components for the onsite and TS groups, respectively. Cronbach's α was 0.76 (onsite) or 0.78 (TS). Lowest versus highest mNRP quartiles differed significantly in both formats (<i>p</i> < 0.001 for each), supporting construct validity. IRR ranged from 0.5 to 0.9, supporting moderate to good agreement between novice and expert raters.Findings provide evidence supporting the reliability and validity of the mNRP tool when applied in both the in situ and TS settings. IRR was acceptable for expert and novice evaluators. This analysis provides additional validity evidence for the mNRP when used to evaluate interprofessional team performance in both onsite and TS formats. · Neonatal resuscitation is a high-acuity, low-occurrence event.. · Many rural clinicians feel underprepared for high acuity, low occurrence events.. · Simulation-based training can improve preparedness.. · Performance is assessed using the mNRP tool.. · Psychometric features support the tool for use in onsite and TS formats..