A mixed-methods approach for complex intervention specification and evaluation (MACISE): illustrative example of Perintatal Psychiatry Access Programs

Aims Complex interventions integrate several core components that interact in ways that are hypothesized to produce the intervention effect(s). Drawing on established practices in program evaluation, we developed the mixed-methods approach for complex intervention specification and evaluation (MACISE) to estimate programmatic reach of complex interventions. Methods MACISE provides a sequential, mixed-methods approach to estimate programmatic reach of complex interventions. First, researchers conduct key informant interviews with intervention team members and review programmatic documents to: (1) characterize the key elements of the complex intervention, (2) elucidate how the core components generate the hypothesized intervention effects (conceptual model) and (3) describe the relevant activities and decision-points in a process map (operational model). Researchers then triangulate and visually depict how variables from the relevant intervention data align with the conceptual and operational model. Next, researchers validate the complex intervention, conceptual model, and process map with program staff through member-checking. Finally, statistical evaluation of programmatic reach of the complex intervention is conducted. Application of MACISE to Perinatal Psychiatry Access Programs (Access Programs) in Massachusetts and Washington State resulted in the development of conceptual and operational models and alignment to the measures for programmatic reach in the respective administrative datasets. Results In both Access Programs, respondents reported the telephone consultation as the “engine” of the respective programs although programs operationally defined consultation differently. Following implementation of the Access Program in Massachusetts (July 2014 to January 2019), the mean number of telephone encounters (operationally defined as a physician-to-physician telephone consultation) for the Massachusetts program was 211.6 (range: 166–272) per month and the mean number of unique providers receiving this consultation per month was 117.0 (range: 88–151). For the 3 years following implementation of the Access Program in Washington (January 2016 to December 2018), the mean number of telephone encounters, (operationally defined as the number of staff-to-physician and physician-to-physician consultations) was 8.6 (range: 3–22) monthly and the mean number of unique providers contacted monthly was 7.6 unique providers (range: 2–21). Conclusion MACISE provides a practical approach to link intervention data to conceptual and operational models to facilitate systematic and accurate estimation of complex intervention reach.