When different methodologies yield different results: reflections on conducting an integrated mixed methods review

With high rates of infants being removed at birth by statutory child protection services across Anglophone countries, the extent to which pregnant people engage with support services to ameliorate the risk of harm posed to infants, and to reduce the likelihood of infant removals, is important. The issue of parental engagement with support services has been examined across a large number of studies and, as such, our team conducted a mixed methods review to synthesize the evidence on the barriers and facilitators to engagement with support services by pregnant people at risk of child removal.1 As our review focused on barriers and facilitators to engagement, we followed a convergent integrated approach.2 This approach calls for quantitative findings to be “qualitized” by providing a short textual description of the findings to support their integration and synthesis with qualitative findings.2 Our review included quantitative studies, both randomized controlled trials and quasi-experimental designs, to test the effects of interventions aimed at facilitating the engagement of pregnant people with support services; survey designs (cross-sectional) to explore reported barriers to and facilitators of engagement; and simulated patient paradigms to explore barriers to service access. When integrated with the qualitative findings extracted from interviews and focus groups with pregnant women and clinicians, this diversity in methodology provided a nuanced understanding of the barriers and facilitators pregnant people face in engaging with support services. However, the integration of these different methodologies also highlighted inconsistencies in findings that could be related, in some instances, to the study methodologies used. The benefits of an integrated approach In some cases, we were able to look to the qualitative findings to provide additional depth and explanation of the quantitative results. For example, the 3 randomized controlled trials included in the review tested the effectiveness of a small-scale monetary voucher as a facilitator to engagement and found these to be ineffective.3-5 Upon consideration of the significant real and/or anticipated consequences of service engagement that pregnant women reported qualitatively (eg, threat of child removal, fear of violent retribution from partners), we were able to better explain why a small-scale monetary voucher may have no impact in this context. Similarly, the quasi-experimental study included in the review found that an information session on mental health and treatment acted as a facilitator.6 This was consistent with, and further explained by, the qualitative reports of pregnant women who described a sense of increased competence and motivation to engage when clinicians explained support options and the capacity for recovery with them. The challenges with survey design studies Inconsistencies in findings arose when comparing the findings of cross-sectional survey studies with the qualitative findings, and such inconsistencies raised questions about the utility of survey designs without the option of free-form reporting. For example, of the 4 cross-sectional studies that used surveys to explore the extent to which stigma was a barrier for pregnant people, all of the studies used items that measured societal stigma (eg, “social stigma,” “fear of neighbors’ negative opinion”).7-10 This contrasts with the experiences of stigma reported qualitatively by pregnant women, who appeared more concerned with experiences of stigma, judgment, and racism directed at them from clinicians within support services. As these barriers lie within the control of clinicians who are well placed to implement research findings, such a distinction regarding the source of stigma within research is important. In the absence of any free-form reporting option, survey design studies also risk missing the issues of most relevance to the population they are studying. We found this to be the case in our review.1 For example, most survey design studies of barriers to engagement included items that measured more general, instrumental barriers, such as cost and transportation issues, and none included the items commonly reported by pregnant women qualitatively, such as fear of child removal and criminal prosecution. When we considered that the items included within survey design studies, and, therefore, the findings of these studies, were not barriers or facilitators voluntarily identified by pregnant women themselves, it became important for us to consider the extent to which participants endorsed each item, rather than to consider the items as stand-alone findings. Therefore, we extracted and included information about the level of endorsement of items in our synthesized findings. We also organized our review findings in a way that prioritized the qualitative findings, and then reported instances where findings extracted from survey design studies were consistent with, or missed aspects of, what women reported qualitatively. Takeaways Conducting an integrated mixed methods review allowed for the integration of a range of study methodologies that provided a full and nuanced understanding of the barriers and facilitators pregnant women face in accessing support services. However, it also uncovered issues with the extent to which the interventions hypothesized to provide benefit (in the case of the small monetary voucher studies) and the measures used (in the case of survey design studies) reflected the lived realities of the those being researched. The risk in this context is that research findings are obscured in a way that suggests a primacy of instrumental issues, such as monetary incentives, cost, and transport, over issues such as stigma, judgment, and racism from clinicians, and fear of child removal. This highlights the benefits of co-design in research and the inclusion of free-form responses in survey design studies. Acknowledgments Mikaela Cibich, Sonia Hines, and Lydia Woodyatt for their work on the mixed methods systematic review.