A survey of how biology researchers assess credibility when serving on grant and hiring committees

Researchers who serve on grant review and hiring committees have to make decisions about the intrinsic value of research in short periods of time, and research impact metrics such Journal Impact Factor (JIF) exert undue influence on these decisions. Initiatives such as the Coalition for Advancing Research Assessment (CoARA) and the Declaration on Research Assessment (DORA) emphasize responsible use of quantitative metrics and avoidance of journal-based impact metrics for research assessment. Further, our previous qualitative research suggested that assessing credibility, or trustworthiness, of research is important to researchers not only when they seek to inform their own research but also in the context of research assessment committees. To confirm our findings from previous interviews in quantitative terms, we surveyed 485 biology researchers who have served on committees for grant review or hiring and promotion decisions, to understand how they assess the credibility of research outputs in these contexts. We found that concepts like credibility, trustworthiness, quality, and impact lack consistent definitions and interpretations by researchers, which had already been observed in our interviews. We also found that, in our sample, assessment of credibility is very important to the majority (90%, 95% CI [87-92%]) of researchers serving in these committees but fewer than half of participants are satisfied with their ability to assess credibility. This gap between importance of an assessment and satisfaction in the ability to conduct it was reflected in multiple aspects of credibility we tested, and it was greatest for researchers seeking to assess the integrity of research (such as identifying signs of fabrication, falsification, or plagiarism), and the suitability and completeness of research methods. Non-traditional research outputs associated with open science practices-research data, code, protocols, and preprints-are particularly hard for researchers to assess, despite the potential of Open Science practices to signal trustworthiness. A substantial proportion of participants (57% [52%, 61%] of participants) report using journal reputation and JIF to assess credibility of research articles and outputs, despite journal reputation and JIF being proxies for credibility that rely on characteristics of research outputs that are extrinsic, rather than intrinsic, to the output itself. While our results only describe the practices and perspectives of our sample, they may suggest opportunities to develop better guidance and better signals to support the evaluation of research credibility and trustworthiness-and ultimately support research assessment reform, away from the use of proxies for impact and towards assessing the intrinsic characteristics and values researchers see as important.