The Reliability and Validity of the ARIS Broader Impacts Rubric

More than 27 years have passed since the National Science Board identified Broader Impacts as one of two merit criteria for National Science Foundation proposals. Yet many researchers remain less certain of how to develop, implement, and assess a broader impact plan. This multi-method study of a Broader Impacts (BI) rubric analyzed expert panels that included BI professionals and researchers for both content validity and reliability. Focus groups with researchers explicate the challenges researchers face regarding BI plans and the potential value of the rubric as a tool for use. It revealed the challenges researchers have in weighing proven strategies versus innovative strategies, a bias documented by other scholars. Researchers stated concern with how to weigh the different facets of the rubric to arrive at a single score. Moreover, researchers reported that their disciplinary field influenced how they interpreted the audiences whose needs and interests may be met through BI plans. These distinctions represent a range of different types of community engaged scholarship (e.g., public information network, community-campus partnership, K-12 schools’ partnerships). Finally, researchers found the BI rubric useful in evaluating and developing their own BI plans as well as their role in panels to ultimately strengthen the field of funded BI work.