Combining Text Mining and Manual Thematic Analysis to Understand Participant Experiences With Surveys

Participant engagement in large-scale surveys is vital to health and social science research. Understanding the factors that affect participant engagement requires methods that can meaningfully interpret participant feedback. Open-ended survey questions can capture these kinds of nuanced responses, but can be challenging to analyze at scale. To address this challenge, this study combines computational text mining with a manual thematic analysis to examine over 15,000 open-text survey responses. These responses provided details about participants’ experiences with longitudinal public health surveys administered by Alberta’s Tomorrow Project. The text mining analysis consists of both sentiment analysis and topic modelling to identify broad patterns in participant experiences. This is combined with a thematic analysis of a purposive sample of 852 of these responses to validate the computational methods and identify themes not captured by the topic model. The findings suggest that many participants found the survey easy to complete and valued having the opportunity to participate. Challenges experienced included difficulty with recall-based questions, irrelevant response options, and program functionality issues. Providing customizable communication methods to adapt to individual preferences was also widely reported to be important by participants. We find that the conclusions from the analyses were largely consistent across the computational text mining and manual thematic analyses. However, the computational approaches overgeneralize or miscategorize certain responses, highlighting the benefits of incorporating both methods to analyse large volumes of qualitative survey responses.