Real-World vs. Trial Engagement: Comparing User Behavior in a Digital Mental Health Intervention

Introduction: Digital Mental Health Interventions (DMHIs) promise to increase access to evidence-based care at scale by circumventing myriad barriers to engaging in mental healthcare (e.g., cost, stigma, and other attitudinal and structural blockers). However, DMHIs are often designed with and for populations who readily participate in consented research trials. Unfortunately, when these carefully designed DMHIs are implemented at scale in real-world settings, they have limited uptake and engagement. To understand the true engagement gap of DMHIs designed in collaboration with consented research participants relative to real-world use, we deployed an 8-week automated DMHI, designed to treat affective symptoms, called Small Steps SMS via a consented randomized controlled trial (RCT) and as a publicly available tool (via a waiver of consent) hosted on a widely accessed mental health community-based organization’s website. Methods: We recruited 80 US-based young adult participants (18-24) who experienced clinically impairing depressive or anxiety symptoms to engage in a consented RCT of Small Steps SMS. We also publicly deployed the same Small Steps SMS intervention (n=5052) under a waiver of consent on Mental Health America’s website. The intervention was made available to any US-based adult (18+) who chose to register for the program and agreed to Mental Health America’s privacy policy. Results: Participants in the RCT were on average younger (Mage=21.6; SD=3.0) and had lower levels of distress (Mk10=27.3: SDk10=7.3) than individuals using the publicly deployed version of Small Steps SMS (Mage=36.8; SDage=12.7; Mk10=33.8; SDk10=8.1). Participants in the RCT actively engaged with the study intervention (i.e., sending replies) 31.6% longer than individuals in the public deployment of Small Steps SMS (PD: n=5052, Mdays=38.6; SDdays=21.1; RCT: n=80, Mdays=50.8; SDdays=11.0). Individuals in the RCT responded 362% more often than did individuals in the publicly deployed version of Small Steps (PD: n=5052, Mresponses=15.89; SDresponses=25.81; RCT: n=80, Mresponses=73.4; SDresponses=39.4). Individuals in the RCT also had 84.7% longer responses (as measured by average characters) to open-ended questions sent by Small Steps SMS (PD: n=2388, Mchar=51.1; SDchar=7.9; RCT: n=80, Mchar=94.4; SDchar=72.5). Conclusions: Participants in consented research trials may not use DMHIs in the same way as real-world users. There is a tension between co-designing DMHIs to meet the needs of consented research participants in formative and summative user centered design studies, and the needs of highly heterogeneous real-world users. Building quality DMHIs for real world deployment requires investment in strategies that increase motivation for change and can address the needs of heterogeneous real world users Limitations: This work constitutes a secondary analysis. Neither the RCT nor public deployment study was designed to test relative engagement between consented research and non-consented real-world DMHI use. Differences with regard to study populations (e.g., age, and initial symptom severity) may also hold relevance for the results presented.