Identification of concepts of importance for the assessment of internal validity of in vitro toxicology studies using a modified Delphi technique

Context: In vitro toxicity studies are increasingly being included as evidence in systematic reviews and chemical risk assessments. INVITES-IN, a tool for assessing the internal validity of in vitro studies, is under development. The first step in the creation of INVITES-IN was the development of an “item bank” database of 405 concepts (“items”) of potential relevance for assessing the internal validity of in vitro toxicity studies. The items were gathered from focus groups and a purposive literature sample that included one systematic review and study appraisal tools for human, animal, and in vitro studies. In this paper we present the second step of the creation of INVITES-IN, i.e. the methods and results for identifying items for consideration when assessing the potential for bias in an in vitro study. Method: A two-round digital Delphi survey, followed by an online workshop with guided discussions, was performed. The Delphi participants were experienced with both in in vitro models and systematic review methods. Results: Fifteen experts completed both Delphi rounds, and thirteen participated in the workshops. Of the 405 items in the bank, 372 were included for consideration when assessing potential for bias in an in vitro study. Both the literature sources and the focus group discussions contributed with concepts considered to be important for the assessment of the potential for bias in an in vitro study; 83%-100% of the items collected from the different literature sources were identified to be important and 91% (127) of the new items discovered in the focus group discussions were identified to be important. Discussion: The 372 retained items will be interpreted into a manageable set of study appraisal criteria and a supporting guidance that will constitute the INVITES-IN study appraisal tool. The high retention of items included in tools designed for assessment of human and animal studies to in vitro studies suggests that many items are generally applicable across multiple study designs. Therefore, tool development processes should benefit from drawing on assessment tools outside the immediate domain of interest. Tool development would also likely benefit from supplementing literature reviews with focus group discussions, as our results demonstrate that the use of focus group discussions with domain experts is a pragmatic and valuable approach to increasing coverage of items in a tool development process. In conclusion, this study demonstrates the value of using rigorous methods to ensure a comprehensive dataset as the starting point for creation of an assessment tool.