Evaluating different scoring algorithms to assess communication ability for individuals living with Angelman syndrome: the Observer-Reported Communication Ability (ORCA) measure

BACKGROUND: Many neurodevelopmental disorders are associated with significantly impaired communication that impacts the individuals and their families' daily activities. The objective of this study is to evaluate the effectiveness of multiple scoring algorithms for the Observer-Reported Communication Ability (ORCA) measure in assessing communication abilities in individuals living with Angelman syndrome and other neurodevelopmental disorders with severe language impairments. METHODS: This secondary data analysis used caregiver-reported data from the ORCA validation study (n = 249) and the Angelman Syndrome Natural History Study (n = 165). Performance of the scoring algorithms was evaluated based on item response theory information functions, the percentage of scores at the floor and ceiling, and the Pearson correlation of scores between the ORCA measure and the Communication and Symbolic Behaviors Scale (CSBS). RESULTS: The ORCA measure's scoring algorithms, which include both emerging and mastery levels of communication behaviors ("Emerging & Mastery"), captured lower levels of ability with greater precision than the algorithm that only assesses mastery levels of behaviors. For overall communication ability in the validation study, the Emerging & Mastery algorithm had floor and ceiling effects of 0.4% each, while the Mastery algorithm had effects of 1.2% and 0.0%, respectively. Both ORCA scoring algorithms for overall communication had strong correlations with the CSBS (r = 0.85 and 0.83, respectively). Similar results were found when scoring ability at the expressive, receptive, and pragmatic (social) communication levels. CONCLUSIONS: Selection of the appropriate ORCA scoring algorithm for use in research studies should be based on the communication concept of interest for the intervention being evaluated and the characteristics of the neurodevelopmental disorder population being assessed. This study reinforces the need for robust assessment tools that can adapt to the unique challenges of rare disease populations.